I work in this field, but nothing state of the art (relatively simple LSTM models and GAN models). While I found the article informative, it was also a little depressing to see how far research goes beyond what I am working on. I spend about 8 hours a week off-work-hours studying and reading papers and I find it difficult to keep up.
I think it makes more sense to focus on the benchmarks. Benchmarks change less often than the underlying algorithms/models, and they are easier to follow.
Once a model performs consistently well on a given benchmarks over several years, then it makes sense to get more into the details.
For example in the NLP field, in 2018, there is a focus on multi-tasks models. Some studies (don't have the refs at hand, sorry) suggest that different models generalize differently (and some time better) when trained on several tasks at once.
Anyway, those papers and models are the result of team of researchers working on the problem full time, with tons of data at their hands. If any sane individuals were able to keep up with the state of the art, it wouldn't be a research field, I guess :-)
Although a neat thought, are the number of papers on ArXiv really something worth comparing to Moore's law? Like at that point, what can't you compare to Moore's law...
IMHO, the two are incomparable. The exponential growth rate of Moore's Law was driven largely by a linear rate of shrinkage in 2D, which drove up clock rates geometrically as microarchitecture component distances shrank two-fold (until CMOS' heat finally fought back). ML has no similar geometric driven basis that will continue to drive its rate of growth superlinear.
I suspect this plot is Dean's way of paying homage to Patterson, since he and Hennessy were famous for similar plots describing CPU performance in their two architecture textbooks.
Even more, an exponent increase in the number of papers on a subject might mean something but it could easily mean something other than the optimistic scenario.
Google really seems to be leading the pack with investments in silicon (e.g. TPU and recently announced edge tpu[1]). Other traditional silicon companies (Nvidia, Intel) seem to get it but I have yet to see investments from other tech companies (Amazon, FB, Netflix).
Microsoft Research is pushing along as well, they just don't publicize it as much.
Azure has already deployed FPGAs (they believe, being able to deploy and make changes including changes to workload on the fly is more beneficial than the efficiency compared to using an ASIC) for networking and accelerated ML (Project Catapult and Project Brainwave).
tbh, I do agree with using FPGAs over ASICs given the speed at which the tech is moving. Google has already cycled through 3 versions of the TPU.
I heard that Google was actually late to the game on this, not realizing the value of GPUs over (just) distributed computing when the DNN trend started. Now they even have custom silicon...
Interesting. There is a kind of obvious conflict between cloud resources and hardware intensive applications like ML. The zeitgeist is obviously swinging.
First, -1 to authors for publishing paper in IEEE Macro which is behind paywall. We need to start mass boycott of all IEEE journals considering they are as bas as Elsevier but have successfully painted themselves as good guys. Number of papers in ML that I came across and behind paywall are mostly from IEEE. In any case, we expect better from authors in Google Brain to agree to publish behind any paywall!
Second, I fail to see any real takeaway or key new insight. Number of papers grows exponentially in many fields in initial periods.
[+] [-] mark_l_watson|7 years ago|reply
[+] [-] antpls|7 years ago|reply
Once a model performs consistently well on a given benchmarks over several years, then it makes sense to get more into the details.
For example in the NLP field, in 2018, there is a focus on multi-tasks models. Some studies (don't have the refs at hand, sorry) suggest that different models generalize differently (and some time better) when trained on several tasks at once.
Anyway, those papers and models are the result of team of researchers working on the problem full time, with tons of data at their hands. If any sane individuals were able to keep up with the state of the art, it wouldn't be a research field, I guess :-)
[+] [-] jahjaylee|7 years ago|reply
[+] [-] spullara|7 years ago|reply
[+] [-] zeusk|7 years ago|reply
[+] [-] randcraw|7 years ago|reply
I suspect this plot is Dean's way of paying homage to Patterson, since he and Hennessy were famous for similar plots describing CPU performance in their two architecture textbooks.
[+] [-] joe_the_user|7 years ago|reply
[+] [-] kingvash|7 years ago|reply
[1] https://techcrunch.com/2018/07/25/google-is-making-a-fast-sp...)
[+] [-] zeusk|7 years ago|reply
Azure has already deployed FPGAs (they believe, being able to deploy and make changes including changes to workload on the fly is more beneficial than the efficiency compared to using an ASIC) for networking and accelerated ML (Project Catapult and Project Brainwave).
tbh, I do agree with using FPGAs over ASICs given the speed at which the tech is moving. Google has already cycled through 3 versions of the TPU.
[+] [-] seanmcdirmid|7 years ago|reply
[+] [-] typeformer|7 years ago|reply
[+] [-] typeformer|7 years ago|reply
[+] [-] drewmassey|7 years ago|reply
[+] [-] ratsimihah|7 years ago|reply
[+] [-] thelastidiot|7 years ago|reply
[+] [-] sytelus|7 years ago|reply
Second, I fail to see any real takeaway or key new insight. Number of papers grows exponentially in many fields in initial periods.