Neural Networks & Backprop

a = (w· x + b), trained by the chain rule.

The idea

A neural network is a function for prediction, built by wiring together simple units called neurons.

A neuron forms the weighted sum $w \cdot x + b$ of its inputs and passes the result through a fixed nonlinear function $\phi$, the activation: $a = \phi(w \cdot x + b).$ With two inputs this reads $a = \phi(w_{1}x_{1} + w_{2}x_{2} + b)$: each input $x_{i}$ arrives on an edge weighted by its own $w_{i}$. The weights $w$ and the bias $b$ are the parameters training adjusts. A layer is a row of neurons reading the same inputs, and a network is layers stacked so that one layer's outputs are the next layer's inputs.

The activation is essential: without it each layer would be a linear map, a composition of linear maps is again a single linear map, and a stack of any depth would express no more than one layer does. A common choice is $\phi(z) = \max(0, z)$, the ReLU, which passes positive values through unchanged and sends negative ones to $0$.

Training minimizes a loss over all the weights by gradient descent, which needs the derivative of the loss with respect to each weight. A weight affects the loss only through the operations downstream of it, so the chain rule applies, and nearby weights share most of the factors in that chain. Backpropagation exploits the sharing.

Algorithm.

Algorithm: Backpropagation Input: a network of layers 1, ..., n with its weights and biases; one training example with its target output Output: the derivative of the loss with respect to every weight and bias 1. ℓ = 1; feed the training example to layer 1 as its input 2. for each neuron of layer ℓ, compute z = w·x + b and a = φ(z); store both 3. if ℓ < n, set ℓ = ℓ + 1, take layer ℓ-1's activations as layer ℓ's inputs, go to step 2 // the forward pass 4. compute the loss from layer n's output and the target, and its derivative with respect to that output 5. ℓ = n 6. combine that derivative with the stored values to get, by the chain rule, the loss derivatives for layer ℓ's weights, biases, and inputs // the derivative for layer ℓ's outputs is known when this runs 7. if ℓ > 1, set ℓ = ℓ - 1 and go to step 6 // layer ℓ's outputs feed layer ℓ+1, so step 6 just computed their derivative 8. return the collected derivatives // the gradient of the loss

Ways to work on it

• Walkthrough. Push an input through a small network of neurons and see how training adjusts the weights.
• Practice. Compute a single neuron's output from its weights and input.
• Hardest. Compute the training gradient of a one-neuron network and diagnose when it vanishes.

Not sure where to start? Take the ten-question placement test.