Backpropagation uses chain rule for gradient-based weight updates

Backpropagation uses chain rule for gradient-based weight updates

  • ◉ AI Geek Programmer
  • ◷ 22 November 2026

A neural network can make a prediction, measure its error, and still have no idea how to improve. The problem this lesson answers is simple: how does a network find which weights caused the error, and how does it change them?

The answer is backpropagation. It uses the chain rule from calculus to calculate gradients. An optimizer then uses those gradients to update the network weights.

From prediction to weight update

A neural network contains weights. These weights control how strongly one input affects the next layer. During training, the network adjusts them so its predictions become closer to the correct answers.

The training loop has four main steps:

  1. Run the input through the network.
  2. Measure the prediction error.
  3. Calculate how each weight affected that error.
  4. Change the weights in a direction that lowers the error.

The first step is called the forward pass. Data moves from the input layer to the output layer. Each layer performs calculations, applies an activation such as ReLU, and passes its result forward.

The network then compares its output with the correct label. For classification, this often uses cross-entropy loss. The loss is a number that describes how wrong the prediction was.

A small loss means the prediction was close. A large loss means the network needs a larger correction.

Backpropagation handles the third step. It starts at the loss and moves backward through the network. At each operation, it calculates how much that operation contributed to the final error.

This is where the chain rule matters. A neural network is a long sequence of connected functions. The output of one layer becomes the input to the next. The chain rule connects the small changes in each part to the change in the final loss.

For a simple composition, the rule looks like this:

[ \frac{dL}{dx} = \frac{dL}{dy} \times \frac{dy}{dx} ]

Here, (L) is the loss, (y) is an intermediate value, and (x) is an earlier value. The full gradient comes from multiplying local derivatives together.

Backpropagation repeats this process across every layer. It reuses intermediate results instead of calculating the same effects again and again. That makes gradient calculation practical for large networks.

A small example

Consider a tiny model with one input:

[ z = wx ]

The model predicts (z). Suppose the input is (x = 2), the weight is (w = 3), and the target value is (10). The prediction is:

[ z = 3 \times 2 = 6 ]

Use a squared error loss:

[ L = \frac{1}{2}(z - 10)^2 ]

The prediction is too low, so the loss is positive. The question is this: should the weight become larger or smaller?

Apply the chain rule:

[ \frac{dL}{dw} = \frac{dL}{dz} \times \frac{dz}{dw} ]

The first part is:

[ \frac{dL}{dz} = z - 10 = -4 ]

The second part is:

[ \frac{dz}{dw} = x = 2 ]

So the gradient is:

[ \frac{dL}{dw} = -4 \times 2 = -8 ]

The negative sign carries useful information. Increasing the weight should reduce the loss in this example.

Gradient descent updates the weight with this rule:

[ w_{\text{new}} = w - \eta \frac{dL}{dw} ]

The symbol (\eta) is the learning rate. It controls the size of each update.

If the learning rate is (0.1), the new weight becomes:

[ w_{\text{new}} = 3 - 0.1(-8) = 3.8 ]

The weight rises from (3) to (3.8). The next prediction becomes (7.6), which is closer to the target of (10).

This example is small enough to calculate by hand. Real networks contain millions or billions of parameters. The same idea still applies. Automatic differentiation systems calculate the gradients for every weight in the computation graph.

Backpropagation calculates the direction. Gradient descent takes the step.

What this means for CNNs

A convolutional neural network, or CNN, applies the same training method to image data.

An image begins as numbers. A grayscale image stores pixel values from 0 to 255. A color image usually has red, green, and blue channels. These numbers have no built-in concept of a face, road, or cat.

A convolutional layer scans the image with small filters called kernels. Common filter sizes include (3 \times 3) and (5 \times 5). At each location, the filter computes a dot product with a local image region. The results form a feature map.

Early filters can learn patterns such as edges and corners. Later layers combine those patterns into textures, shapes, and object parts. The final layers produce class scores or probabilities.

Several filters can scan the same image at once. Each filter can learn a different pattern. The same filter is reused across the image, a design called parameter sharing. This keeps the number of weights lower than in a fully connected network that treats every pixel as unrelated.

Stride controls how far a filter moves at each step. A stride of one checks every nearby position. A larger stride skips positions and reduces the output size.

Padding adds values around the image border. It helps preserve spatial size and gives edge pixels a fair chance to affect the result.

Pooling layers reduce feature-map size. Max pooling keeps the highest value in a small region. Average pooling calculates the region’s mean value. Pooling reduces later computation and can make predictions less sensitive to small shifts in an image. It also removes some detail, so it is not free.

A common CNN structure repeats convolution, ReLU, and pooling. Later layers flatten the feature maps and pass them to fully connected layers. Backpropagation then moves from the classification loss through those layers, through the convolution operations, and into the filter weights.

The filters learn because their gradients show how each filter affected the final prediction.

Making optimization behave

A correct gradient does not guarantee useful training. The optimizer still needs sensible conditions.

The learning rate is one of the main controls. If it is too large, updates can overshoot and make the loss unstable. If it is too small, training can move at a painfully slow pace.

A learning-rate schedule changes the rate during training. Some schedules reduce it over time. Warm restarts raise it again at planned points, which can help the optimizer explore new regions.

Batch normalization can help keep activations within a more stable range during training. Stable activations can make optimization easier, though batch normalization also adds behavior that depends on training and evaluation modes.

Gradient clipping limits very large gradients. This helps when one update could cause a large and harmful weight change. Clipping does not repair every training problem, but it can prevent exploding updates from taking over.

Validation loss gives a view of performance on data the model did not use for the current weight updates. If training loss keeps falling but validation loss starts rising, the model may be overfitting. Early stopping can end training before that gap grows too far.

These techniques are control tools, not magic settings. Their value appears through measurement. Training loss, validation loss, learning rate, and gradient behavior reveal whether the model is learning or merely moving numbers around.

Backpropagation is the engine that turns error into gradients. The chain rule connects each local calculation to the final loss. Gradient-based optimization then uses those gradients to change the weights.

That gives a practical foundation for understanding CNN training. A CNN learns visual features from pixel values because its filters receive feedback through the same backward process. The next step is to study how those filters turn raw image grids into useful visual representations.

The Model Log keeps that learning path grounded in one practical AI concept, one working example, and one honest look at what actually works.

Tags:
    Share:

    Related articles

    Cloud computing runs on distributed server networks

    Cloud computing runs on distributed server networks

    • AI Geek Programmer
    • 22 November 2026

    Cloud computing runs on distributed server networks.

    Read article
    AI architecture requires zero trust security models

    AI architecture requires zero trust security models

    • AI Geek Programmer
    • 21 November 2026

    I keep coming back to one plain fact: AI systems do not live behind one clean wall anymore.

    Read article
    AI systems reduce software development agency costs

    AI systems reduce software development agency costs

    • AI Geek Programmer
    • 20 November 2026

    AI systems reduce software development agency costs. That is the plain answer, but the real story is narrower than the hype.

    Read article