[zump]

The key to understanding is drilling the backpropagation algorithm and being able to visualize the application of the multivariate chain rule as a computational graph.

EDIT: You won't understand until you do this yourself using pen and paper. It's a pain.

EDIT2: This nuts and bolts tutorial will help

http://briandolhansky.com/blog/2013/9/27/artificial-neural-n...

[chestervonwinch]

The derivation is a pain -- there's a lot of notation and indexes to keep track of.

It might be an easier first step for someone starting out to derive the gradient terms for the cost function for logistic regression since it can be viewed as a classification neural net without the hidden layer(s).