Week 4 — Backpropagation: learning across all the network's parameters

Week 2 explained how gradient descent adjusts a parameter.

Week 3 organized many weights and biases into neurons and layers, leaving four questions:

How much should w₁ change?

Should w₂ increase or decrease?

How sensitive is the final loss to a particular weight?

How does a weight near the beginning receive a learning signal from loss?

Backpropagation provides the gradients needed to answer these questions.

This week focuses on four ideas:

The central idea:

Backpropagation starts at loss and follows computational dependencies backward, using the chain rule to calculate how each parameter affects loss locally.

The complete training sequence remains:

Forward Pass

↓

Prediction

↓

Loss

↓

Backpropagation

↓

Gradients

↓

Optimizer / Gradient Descent

↓

Updated Parameters

Within that sequence:

Keep these two jobs separate.

Consider a very small network:

Its dependencies are:

x, w₁, b₁

↓

z

↓ReLU

h

↓with w₂, b₂

ŷ

↓compare with y

L

This is a computational graph: split a large calculation into small operations and record which results depend on which inputs.

For a programmer, it resembles a runtime dependency graph.

Why split the formula into a graph?

In principle, we could differentiate a huge expression for the whole network, but maintaining it would be difficult. A tensor may feed several layers, and a parameter may affect loss through several paths. A computational graph breaks this into repeated small questions:

Which inputs does this operation receive?

What output does it produce?

Which later operations use that output?

What are its local derivatives?

This resembles a compiler's intermediate representation or a service dependency graph. Each node describes its own behavior, while the graph records the dependencies. The same chain-rule traversal principle applies as the number of operations grows.

What is actually saved?

The graph records more than call order. Backward calculation also needs context from the forward pass. For multiplication:

Backward needs the values of a and b used during forward, because:

Retaining required intermediate values trades memory for the information needed by backward. This is one reason training usually requires more memory than inference without gradient tracking.

Start a new, smaller network with one hidden neuron so that we can follow every edge. Do not reuse Week 3's two-hidden-unit values. Record x, w₁, b₁, w₂, b₂, z₁, a₁, prediction, and loss. Use this same old state until the parameter update in Section 11.

Set:

x = 2

y = 5

w₁ = 1

b₁ = 0

w₂ = 1

b₂ = 0

Forward：

Saved values include z, h, and ŷ, which are used again when evaluating local derivatives during backward.

A training forward pass does more than predict

During training, the forward computation does two jobs:

Inference normally needs predictions, not a subsequent backward pass, so gradient tracking can be disabled. For training, the required intermediate information must remain available or be recomputed: backward needs to know whether ReLU's input was positive or negative and which forward values belong in a multiplication derivative.

Backpropagation does not differentiate the whole expression in one leap. Each node answers:

If one input changes slightly, how quickly does my output change?

More precisely, a local derivative answers:

Within this operation, how sensitive is the output to a small change in one input, with other inputs held fixed?

“Local” means the operation does not need to understand the entire network or know the final loss value. Multiplication handles multiplication; ReLU handles ReLU. Combining these local relationships along the graph gives the sensitivity of the final loss to a parameter.

Think of each node as a component implementing a common interface:

forward(inputs) → output

backward(upstream_gradient) → input_gradients

Forward produces values. Backward receives a gradient from later computation, combines it with local derivatives, and propagates contributions to the inputs.

Some common local derivatives:

Addition

Why are both derivatives 1? For c=a+b with b fixed, increasing a by 0.01 increases c by exactly 0.01. The change multiplier is 1. An addition node passes the incoming gradient unchanged to each input; it does not split it in half.

Multiplication

For c=a×b, suppose forward used a=3 and b=4. Hold b fixed and increase a by 0.01:

Originally c=3×4

= 12

Now c=3.01×4

= 12.04

a increases by 0.01 and c increases by 0.04. The multiplier is 4, which is the other input b. This explains the multiplication derivative instead of leaving it as a memorized rule.

Squared error

The derivative with respect to prediction encodes both direction and scale:

Squared error scores the discrepancy. Its derivative tells us which local change in the prediction would reduce that score. Parameter gradients also need the remaining chain-rule factors.

ReLU

At z=0, ReLU has no unique ordinary derivative; an implementation chooses a convention for backward. Our worked example avoids this point.

ReLU's local derivative acts like a gradient gate:

ReLU does not create a gradient from nothing. Its forward input determines how much of the incoming gradient passes through this path.

How a local derivative participates in backward

The core scalar rule for a node is:

For example, the multiplication node c=ab receives:

upstream gradient = ∂L/∂c = -6

Forward used a=3

Forward used b=4

The gradients it sends to its two inputs are:

This is what the local derivative formulas are for: they are not isolated entries in a table. They are the multipliers each node uses to propagate the sensitivity of loss backward to its inputs.

This section temporarily changes a local multiplier to isolate the multiplication-along-a-path idea. That does not update the main experiment's saved parameters from Section 3. Return to that old state in Section 6. The chain rule calculates sensitivities; the optimizer separately determines parameter updates.

Each factor is a local sensitivity between adjacent nodes.

Why multiply? Because:

Change w₁ slightly

That changes z

The change in z changes h

↓

The change in h changes prediction

↓

The change in prediction changes loss

Along this single path, the final local sensitivity is the product of the local sensitivities.

Why multiply rather than add?

Suppose a small change passes through three stages:

Increase w₁ by 0.01

z changes by approximately 2×0.01

h changes by approximately 1×the change in z

ŷ changes by approximately 3×the change in h

Finally:

Each stage scales the change from the previous stage, so the multipliers multiply. The chain rule expresses this composition precisely in terms of derivatives.

A familiar dependency-chain analogy

If:

Tax = Tax(Subtotal(Price(quantity, unitPrice)))

A change in unitPrice affects Tax through Price and Subtotal. Along this path, a zero local sensitivity at any stage makes its first-order contribution to the final change zero. Neural-network backward calculations follow the same dependency idea, with more nodes and branches. The gradients needed for our training loop are first derivatives.

Backpropagation here calculates first derivatives of loss with respect to parameters. It does not first need second derivatives. A nonzero first-order gradient identifies the local direction of increase; moving in the opposite direction with a sufficiently small step can decrease a differentiable loss. An arbitrary finite step is not guaranteed to do so.

Second derivatives describe curvature. Some optimization methods use them, but a full neural-network Hessian can be extremely large. GPT training commonly uses first-order methods such as SGD or AdamW without explicitly constructing it.

The forward pass gives:

ŷ = 2

y = 5

L = 9

The logical starting point for backward is:

The derivative of a value with respect to itself is 1. Seed the loss node with 1, then propagate backward through the local derivatives.

Why usually reduce loss to a scalar? It specifies a single objective and allows a seed of 1. For a vector output, a backward calculation needs a supplied vector of weights to specify a vector–Jacobian product, or an explicitly defined scalar objective. Summing or averaging batch losses gives the scalar objective we use in training.

Next, calculate how loss changes with prediction.

First step:

This −6 is the gradient passed from loss to the prediction node.

It means a sufficiently small increase in prediction decreases loss at the current point.

Output：

For w₂:

Chain Rule：

For b₂:

So:

We now have local loss sensitivities for both output-layer parameters. The update rule will use them later.

To continue, calculate the loss sensitivity to h:

So:

This is a gradient with respect to an intermediate activation, not a parameter. It carries the sensitivity needed to reach earlier parameters.

During forward:

z = 2 > 0

Therefore:

So:

If forward used z<0, ReLU's local derivative is 0, so this path contributes zero gradient.

Hidden linear calculation：

For w₁:

So:

For b₁:

Finally:

gradient_w₁ = -12

gradient_b₁ = -6

gradient_w₂ = -12

gradient_b₂ = -6

We now have gradients for all parameters with respect to the same loss.

No parameter has changed yet. Every gradient describes the loss from the same old forward pass. Set the learning rate to 0.01; only now apply “old parameter minus learning rate times gradient.”

Why not update w2 as soon as dw2 is available? The old forward pass gives dL/dprediction=−6, and propagating to h requires the old w2=1, so dL/dh=−6. Using the new w2=1.12 instead gives −6.72 and then the incorrect dw1=−13.44 after multiplication by input 2. The correct value is −12. The incorrect calculation mixes two parameter states.

θ denotes the four parameters collectively; ∇L is their corresponding gradient vector. Backward calculates those gradients. The optimizer changes parameters according to the chosen update rule. These are separate operations.

The next section recomputes forward: z=1.12×2+0.06=2.30, h=2.30, prediction=1.12×2.30+0.06=2.636, and loss=(2.636−5)²=5.588496. The old variable holding loss=9 does not update itself; we must calculate again.

New parameters:

Compare:

Before: Loss = 9.00

After: Loss ≈5.59

After this backward calculation and update, the prediction has moved toward the target in our example.

Each gradient calculation corresponds to an edge or combination of paths in the graph.

You can check the result without another symbolic derivation: perturb one parameter by +ε and −ε, hold all other parameters fixed, and calculate both losses. The central difference [L(θ+ε)−L(θ−ε)]/(2ε) should be close to the hand-derived gradient. This is a small-example diagnostic, not an efficient way to train a large model parameter by parameter. Avoid ReLU's nondifferentiable zero point. The runnable example is course_examples/week04_gradient_check.py.

A parameter or intermediate value can affect loss through several paths.

The rule:

Add gradient contributions from different paths reaching the same variable.

For example:

Then:

Think of one shared dependency used by several callers: its total effect must include all relevant paths.

Why add here? Each path contributes to the local change. Omitting a path can give the wrong total. Contributions can reinforce each other or cancel because gradients have signs.

Remember two graph rules:

Along one continuous path: multiply local derivatives

When paths lead back to the same variable: add their gradient contributions

“Multiply along paths, add contributions at branches” captures the basic scalar structure of backpropagation.

A batch contains multiple examples:

For a mean loss, the parameter gradient is the mean of the per-example contributions:

The gradient reduction follows the loss reduction. This is why averaging batch losses also averages their gradient contributions.

PyTorch Autograd automates three jobs:

It automates repeated calculations without changing the underlying mathematics.

In its usual eager execution mode, PyTorch builds the differentiation graph from the tensor operations actually executed during each forward pass. Python conditionals, loops, and function calls can therefore determine which operations appear. Ordinary backward normally releases saved intermediate values needed for differentiation; the next training step builds a new graph. Retaining a Python reference to an output is not the same as retaining all saved values needed to run backward again.

This explains several common behaviors:

Along a path through many layers, local derivative factors multiply.

If many factors have absolute values below 1, a path's contribution can become very small: a vanishing gradient.

If many factors have absolute values above 1, a path's contribution can become very large: an exploding gradient.

Later we discuss choices such as ReLU, residual connections, LayerNorm, and gradient clipping that can help particular aspects of training stability. None is a universal guarantee, and detailed proofs are not required here.

A small numerical illustration: passing through twenty local factors of 0.5 gives a total multiplier of:

The contribution becomes tiny. Passing through twenty factors of 2 instead gives:

The issue is not depth alone, but how local sensitivities compose. Repeated products can become extremely small or large. Later architectural and optimization choices address aspects of these effects; a full network can also have multiple paths whose contributions combine.

Derivatives, gradients, and parameter gradients

These terms are easy to mix up in code:

Intermediate tensors also have mathematical gradients, although a framework need not retain them in each tensor's .grad field by default. An optimizer normally updates registered trainable parameters, not activations. Backpropagation combines local derivatives to calculate parameter gradients.

Therefore:

If the learning rate is 0.01:

w_new = w - 0.01 × (-12)

= w + 0.12

We can now calculate a small network's forward pass, backward pass, and parameter update by hand.

Real networks can contain millions or billions of parameters, making manual maintenance of all derivatives impractical.

Week 5 hands the same mathematics to PyTorch:

Python numbers

↓

Tensor

↓

Autograd

↓

nn.Module

↓

Optimizer