Convolutional Neural Networks

Convolution, weight sharing, output sizing, and pooling for grids.

The idea

A convolutional layer is the building block of neural networks that process grid data — signals, images. It slides a small array of learned weights, the kernel, across the input: at each position it multiplies the kernel entrywise against the patch underneath and sums the products. The sums, laid out in grid order, form the feature map.

Two restrictions separate this from an ordinary layer, in which every output is a weighted sum of every input. Each output depends only on a small patch, and the layer applies the same kernel at every position instead of learning fresh weights for each location. So the layer holds only as many weights as the kernel has entries, however large the grid, and whatever pattern the kernel responds to, it detects anywhere on the grid.

A pooling step often follows: it replaces each small neighbourhood of the feature map by a single summary, such as its maximum, so that a slight shift of the input leaves the output nearly unchanged.

The feature map's length is the number of positions the kernel fits. A kernel of size $K$, moved in steps of $S$ across an input of length $W$ padded with $P$ zeros at each end, fits in $\left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1$ positions.

Ways to work on it

• Walkthrough. The convolution operation, the output-size formula, and why weight sharing saves parameters.
• Practice. Apply the output-size formula across kernel sizes, strides, and padding.
• Hardest. Size a 2-D conv layer and count its parameters against a fully connected layer.

Not sure where to start? Take the ten-question placement test.