Intel Gets Serious About Neuromorphic, Cognitive Computing Future

[paulsutter]

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical.

Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs.

Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients).

[1] https://devblogs.nvidia.com/parallelforall/mixed-precision-p...

[2] XNORnet, https://arxiv.org/abs/1603.05279

[3] DoReFa, https://arxiv.org/abs/1606.06160

[deepnotderp]

This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.

[emcq]

For training networks, designing en efficient chip is a scary proposition.

However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs.

I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like.

[0] https://arxiv.org/pdf/1603.08270.pdf

[russdill]

I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.

[anigbrowl]

Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.

[deepnotderp]

XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".

[justinpombrio]

Could someone explain what a neuromorphic chip is?

The article assumes we know, but I haven't heard of it. And the wikipedia article on "neuromorphic engineering" talks about stuff like analog circuits, copying how neurons work, and memristors, none of which seem that related.

[mljoe]

The best definition I can come up with is hardware that implements neural network architecture directly, especially McCulloch–Pitts spiking neurons (which have a temporal component). In neuromorphic chips, neurons are an actual component in the hardware, you can ask questions like "how many neurons does this chip have?". Contrast to how neural nets as we use them today which are actually implemented as a computation graph on tensors. It turns out that a special kind of neural network can be abstracted well as a series of tensor ops (dense feedforward layered networks[1]), but this is not necessarily the case for any neural network. So neuromorphic chips have a possibility of being far more general.

[1]: Which are wired something like this: http://neuralnetworksanddeeplearning.com/images/tikz40.png - Notice that dense connections and layered architecture. For all intents and purposes, this what neural nets look like today because of how easily it is to treat a NN with this specific wiring as a chain of tensor computations and thus execute on more conventional hardware.

[nomailing]

What I remember from my neuromorphic engineering course (or analog VLSI course) is that we designed the silicon layout (with n and p doping regions) in a way that the transistors are operating in the subthreshold regime in the IV characteristics. If I remember correctly the IV characteristic is linear in the subthreshold region? In contast in normal digital chips only the super-threshold region is used (voltage above a certain saturation threshold switches the transistor completely on). Using the subthreshold region it is possible to implement spiking neuons with only very few transistors. It works completely different than digital circuits. The connections between the transistors don't transmit just 0's and 1's. Instead all wires transmit analogue signals where the exact voltage matters. This makes these chips extremely energy and space efficient. These chips can also work much faster even in comparison to biological neurons (obviously using some assumptions and simplifications, such as neglecting certain special kinds of ion channels found in real neurons).

[return0]

Loosely defined, it's various chips that implement simplified computational models of neurons, and some plasticity functions. They are usually simplified because going into the greatest detail (modeling hodgkin-huxley type channels) would require too much computation. In the neuroscience community there is not yet an acceptable model for a simplified neuron so everyone picks some spiking neuron model or they make up one. Even less is known for plasticity functions.

I believe the idea is that if you simulate too many of them, something useful will happen.

[mattlevan]

All of the subjects you mention at the end of your comment, especially memristors, are indeed the focus of neuromorphic computing. It's a nascent technology field, but there are plenty of papers available on IEEExplore or ACM Digital Library to satisfy curiosity! I'll take a look at the survey paper I wrote a couple summers back-- if it's decent I'll edit this post with a link.

Edit, links: Paper: https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdv... Slides: https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBP...

[bobsil1]

Asynchronous, analog, spike-based, co-located memory. Closer NN sim than von Neumann architecture, orders of magnitude more power-efficient.

One human brain-equivalent NN on classic architecture costs ~$70M and uses ~100 houses worth of power.

[emcq]

The near future is not in putting learning inside a chip. We're a long ways off from the one shot learning needed to make a device actually interesting with localized learning.

Instead, the future is recording and uploading your observations to the cloud, with data scientist and neural net wizards training over this dataset on a cluster with tons of GPUs, and then deploying an optimized model to scrappy low precision inference chips.

This is why FPGA based designs will fail to be stunning. Specialized low precision ASICs more similar to DSPs like Movidius' Myriad (in the Phantom drones and Google's Project Tango devices), Google's TPU, upcoming Qualcomm chips, or Nervana's will become increasingly popular.

[espeed]

Are these the Neuromorphic chips [1] Jeff Hawkins of Numenta [2] has been talking about?

[1] Neuromorphic Chips https://www.technologyreview.com/s/526506/neuromorphic-chips...

[2] Numenta papers/videos http://numenta.com/papers-videos-and-more/

It would also be good to see a major chip manufacturer or cloud provider that makes its own chips (Google/IBM) get serious about graph processing chips [3,4] and moving beyond floating point [5].

[3] Novel Graph Processor Architecture https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20...

[4] Novel Graph Processor Architecture, Prototype System, and Results https://arxiv.org/pdf/1607.06541.pdf

[5] Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic https://www.youtube.com/watch?v=aP0Y1uAA-2Y

[m3kw9]

Seem like a fancy name for parallel computing chip that specialize in efficient parallel computing

[adevine]

It's that what it actually is? Not familiar with the field, so is "neuromorphic chip design" really "we wired a bunch of GPUs together?"

Don't mean to minimize the work involved, just trying to decipher the marketing speak.

[return0]

Do 'Neuromorphic' chips do anything useful? At least neural networks have well known utility, but afaik, neuromorphic (i.e. heavily simplified models of neurons that one only hopes - but cannot prove are correct) have no useful applications - or even theoretical functions.

[bobsil1]

Low-power video object recognition for military drones, security cams, etc.

[mrfusion]

Would it be possible to build an analog neural network with hardware?

[emcq]

Yes, this is traditionally a big effort in the neuromorphic community originating in Carver Meads lab at Caltech.

Perhaps the best success of such analog computation efforts are with Neurogrid (full disclosure I worked with this group): https://web.stanford.edu/group/brainsinsilicon/

One thing to keep in mind is that communication is often still in digital spikes, which some may argue is more neuron like than an analog encoding.

[deepnotderp]

Yes, I work at a startup that's doing this, but it's very dangerous with the amounts of noise, which seem fine on MNIST and even CIFAR, but you die on imagenet.

The key to circumventing this is very complicated and our "secret sauce".

[adamnemecek]

I'm hoping that we could see photonic computers.

[aj7]

For laughs, I searched this document for the term Nvidia.