Andrew Ng and the Quest for the New AI

[jhartmann]

I've said this before, but deep learning is terribly powerful precisely because you don't have to spend lots of time doing feature engineering. Multi layer networks that are trained in semi-supervised, unsupervised and supervised fashions all can now produce networks that meet or beat the state of the art hand created models for speech, handwriting recognition, ocr, and object recognition. We are only just beginning to see what is possible with these sorts of techniques. I predict within a few years time we will see a huge renaissance in AI research and Neural Network research specifically as these techniques are applied more broadly in industry. My startup is building some cool stuff around this technology, and I know there are hundreds like me out there. This is going to be a fun ride.

[wookietrader]

This has been said about neural nets two times already. Sadly, they did never deliver.

There are still applications where e.g. random forests beat the crap out of all kinds of deep learning algorithms in (a) training time (b) predictive quality (c) prediction time.

We should stop hyping this. I am a researcher working in deep learning myself, but the current deep learning hype is actually what makes me worry that I will have trouble getting a job because industry will be disappointed a third time.

[wookietrader]

If an article says that Andrew Ng is "the man at the center of deep learning" it's just not right. Geoffrey Hinton's and Yoshua Bengio's impact were at least as high as his, if not much higher.

[kinofcain]

There is a very quick reference to the person who inspired him, Jeff Hawkins, whose book is worth a read:

http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/080507...

Edit: update link

[thomaspaine]

Grok (formerly Numenta) has a slightly more technical white paper that goes into more detail on the actual algorithms from On Intelligence:

https://www.groksolutions.com/htm-overview/education/HTM_Cor...

There are some subtle differences between HTMs and straight up deep learning, mainly the requirement for HTM data to be temporal and spatial.

I know Andrew used to sit on an advisory committee at Numenta, I don't know if he still does.

[netrus]

Link does not load CSS for me, but this does:

http://www.amazon.com/Intelligence-Jeff-Hawkins/dp/B000GQLCV...

[lomendil]

I just saw Jeff Hawkins give a talk and it was quite interesting. I was a bit worried, however, that he is basing his theory of intelligence on the human neocortex, while claiming to go after general principles.

This is guaranteed not to be terribly general, considering the many bits of matter on this planet that exhibit intelligence without a neocortex. By many, I mean ones that hugely outnumber humans.

So very interesting stuff, but not the answer that I think he wants it to be.

[blhack]

That book will change the way that you look at yourself.

[EthanHeilman]

We should stop trying to claim every new method is "like the brain". We don't have any clear understanding of how the brain works. One can be inspired by a particular and likely wrong cognitive theory, but one can not say one is building "machines that can process data in much the same way the brain does" truthfully without a deeper, and currently unavailable, understanding of the functioning of the human brain.

[Jach]

I feel the same way, that researchers should stop trying to mimic the brain, but not because we don't understand the brain. While I think there are still several decades before we'll be able to have mind uploads, I also think a lot of people underestimate the quality of modern brain science. In any case, I have the same reason as Dijkstra for why I think mimicking the brain isn't that great an idea. In http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036... (really a great read to branch all sorts of thoughts off of) Dijkstra said, "The effort of using machines to mimic the human mind has always struck me as rather silly: I'd rather use them to mimic something better."

It's probably a harder problem, creating smarter-than-human intelligence on a machine, but research isn't as constrained by laws and ethics (they don't have to bemoan not being able to experiment with living human brains). I wish more people were active in the area.

[hvs]

OT: His online Machine Learning class last year was great. He is the best professor I've ever had, and explains things so clearly that you understand them the first time. You are lucky if you ever get to work or study under him.

[mark_l_watson]

I second this! I have taken 7 Coursera classes, and most of them "lightly": just doing as much work as I needed to for passing the class, with just a few classes that I did put a lot of energy into. Andrew's class was in this second category: I kept taking the tests and tweaking the homework assignments over and over again until I got a 99.5% score in the class. His class was lots of fun and also very useful material. Recommended!!

[rzendacott]

That's awesome to hear! I'm taking the Coursera class now, and it's been great so far. It just started a couple weeks ago, so it definitely isn't too late to join! https://www.coursera.org/course/ml

[alanthonyc]

It’s great this year as well. Very enjoyable class.

[ivanist]

I have to blame you, because now I had to take the course in coursera! I always wanted to learn machine learning. Better late than never!

[gavanwoolery]

Hmm...I don't mean to be a skeptic, but I do not see any new theories here. Neural networking has been around for a long time, as have an abundance of theories and implementations around it...some people have gone so far as to build an actual brain replica (a digital version of the bio/analog thing). Neural networking is extremely powerful, but to be of any use, you need a lot of computing power. As it turns out, our brains are really good at massively parallel tasks like image and motion processing; these things can be done explicitly on a computer with some ease, but having a computer learn on its own from scratch how to do them is not easy.

[jbelanich]

You're correct in that neural networks as a model have been around for a long time. However, those networks were restricted to be shallow because backpropogation didn't work well on networks with many hidden layers. Only recently have researchers developed learning procedures that can learn these deep architectures efficiently, using some clever unsupervised learning techniques. And surprisingly, they are finding that these deep networks perform remarkably well, beating the state of the art in a number of benchmarks.

You are also right that you do need a lot of processing power to get neural networks to work well. But that is changing rapidly. Hinton's convolutional neural network has the state of the art in the ImageNet benchmark, yet was trained using significantly less power than google brain. Regardless, you don't need google scale computation to get deep networks to work well. The point of google brain is to see how far one could push neural networks.

[raverbashing]

Exactly

Neural networks of course "mimic" the way the brain works, but it stops there.

Not to mention several modern AI techniques have nothing to do with mimicking biology (like SVMs)

This looks a reporter pushing his own agenda to make for a colorful story.

[xtacy]

I think the requirement on enormous computing power is exactly the point. Since not many have access to such computing power, Andrew Ng and folks at Google (incl. Jeff Dean!) built a large scale neural network system called DistBelief (nice name!). This system allows programmers to think of really large scale neural networks without worrying about how to scale them, handling fault tolerance, etc. You can think of it as the MapReduce for neural networks. They demonstrated how a large scale neural network can do interesting stuff on its own (e.g. recognising cats and human faces from unlabeled youtube videos).

[why-el]

So I am a little confused. Where are we on the learning part of AI? As I understand it, the current consensus is to throw as much data as you can at your model (millions of cat pictures in this article's example) to make it pick up patterns and yet still claim that we are closing in on how the brain works? As far as I can tell no human brain would need that many pictures to see a pattern. In fact, and this is probably more apparent in language, we humans tend to work with degenerate data and still end up with perfect models.

[piglop]

You may not have seen millions of cats, but how many images have your brain processed since you can see? A five years old brain has been trained on billions of images (along with many other simultaneous inputs).

[DennisP]

Bear in mind that the neural network is starting at the same point as a newborn baby.

[pilooch]

It seems the relationship between the brain and deep learning has evolved in such a way that the later can help with insights into how the former works.

In this regard, I thought I would mention the extraordinary simple and elegant talk by G. Hinton last summer: http://www.youtube.com/watch?v=DleXA5ADG78

It starts from a simple and clever improvement to an existing deep learning method and ends up with beautiful (and simple!) insights on why neurons are using simple spikes to communicate.

[aespinoza]

I thought the man behind the Google Brain was Ray Kurzweil (http://www.wired.com/business/2013/04/kurzweil-google-ai/).

[drcross]

Kurzweil is a hack. (Source: I've an Msc in Robotics and read all his books)

[cscurmudgeon]

Ah. The good old AI cycle.

Scientist: X can help us get full AI!

You: Why?

Scientist: Because of reason R.

You: But, reason R is a non sequitur...

More seriously, reasons similar to that for deep learning have been repeated multiple times in AI with failure (e.g. Thinking Machines).

I would suggest that these folks remain calm and build something on the scale of IBM's Watson using just deep learning..

[niklaslogren]

Very interesting article, it makes me hopeful.

This might be slightly off-topic, but I'll try it here anyway: can anyone recommend any books/other learning resources for someone who wants to grasp neural networks?

I'm a CS student who finds the idea behind them really exciting, but I'm not sure where to get started.

[yankoff]

Great and inspiring professor. Taking his ML course on coursera and trying to follow his talks.