Week 2

Week 2 — Linear regression: from prediction to gradient descent

Key questionHow does a model use errors to learn its parameters?

Learning objectives

Understand and apply Week 2 — Linear Regression: from Prediction to Gradient Descent

75 min estimated reading time

Week progress: 0 of 35 sections (0%)Course progress: 0 of 321 sections (0%)

In Week 1, a person chose the parameters. This week, the program adjusts them using known answers. We begin with four points: x=[1,2,3,4], y=[3,5,7,9]. They happen to follow y=2x+1, but the training program reads only the data; it does not copy those answer parameters.

Follow four questions: How wrong is the prediction? What happens if we change one parameter slightly? How do we choose a direction and step size? How do we combine suggestions from four samples? To make the arithmetic manageable, we temporarily use just x=2, y=5 with w=1, b=0. When we return to four samples, we explicitly reset w=b=0.

Scroll horizontally to view all columns.

Course data table
Study sessions	What you should be able to do without the book
1: Predictions and loss	Calculate errors and explain why we square and average them.
2: Small changes and derivatives	Calculate L(1) and L(1.001), and explain what −12 means.
3: Partial derivatives and updates	Distinguish updating only w from updating both w and b; calculate every gradient using the old parameters.
4: Repeated learning and diagnosis	Run course_examples/week02_loss_gradient.py and compare three learning rates.

You only need arithmetic and squaring as prerequisites. Second derivatives and the normal equation are optional reading, not prerequisites for writing the training loop. This week ends with a loop that actually changes parameters. Week 3 keeps the same learning principle while making the prediction function more flexible.

1. What problem does linear regression solve?

Scroll horizontally to view all columns.

Course data table
Reading range	Fixed data and starting point	Why this sequence?
Sections 1–10	One-point illustration: model, error, and slope	First distinguish these three objects
Section 11	x=2, y=5, w=1; temporarily hold b=0	Adjust just one control
Sections 13–14	The same point, now with a trainable b	Partial derivatives for multiple parameters; restart the example
Code in Section 17	x=[1,2,3,4]，y=[3,5,7,9]；w=b=0	Mean loss over four samples; not a continuation of Section 14

The rule y=2x+1 is a teaching pattern we deliberately used to generate this week's data. The training program sees only input–target pairs, not the answer parameters. You will observe how errors guide it toward the pattern.

Linear regression predicts continuous numerical values, such as house prices, sales, temperature, or distance.

Suppose we have these historical data:

Scroll horizontally to view all columns.

Example data for linear regression
x (floor area)	y (house price)
1	3
2	5
3	7
4	9

The pattern is:

y = 2x + 1

If x=5, the predicted y is 11.

The central idea of linear regression is to fit a line to available data and use it to predict values for new inputs.

2. Model、Feature、Parameter

Model:

\hat{y} = wx + b

Here:

x: the input feature
y: the true value, or target
ŷ: the predicted value
w: the weight
b: the bias
w and b: parameters the model must find through training

From a programmer's perspective:

python

def predict(x, w, b):
    return w * x + b

Training seeks values of w and b that minimize the chosen training objective.

3. How is a prediction produced?

Suppose:

w = 1

b = 0

x = 2

Then:

\hat{y} = wx + b = 1 \times 2 + 0 = 2

But the true answer might be:

y = 5

So:

Prediction = 2

Actual

= 5

The prediction is wrong.

4. Error versus loss

The simplest signed error is:

\mathrm{Error} = \hat{y} - y

For example:

2 - 5 = -3

The prediction is 3 too low.

Adding signed errors can allow positive and negative errors to cancel. A loss function gives us an objective that avoids this particular problem.

One common choice:

\mathrm{SquaredError} = (\hat{y} - y)^2

For example:

(2 - 5)² = 9

Squaring has two effects:

It removes the sign;
It penalizes larger errors more strongly.

Scroll horizontally to view all columns.

Course data table
Prediction / target	error = prediction − target	Squared error
3 / 5	−2: prediction too low	4
7 / 5	+2: prediction too high	4

The sign of the error tells you whether the prediction is too high or too low. Squared loss measures the size of the discrepancy; opposite errors no longer cancel. If the target is measured in dollars, squared error is measured in dollars². Taking the square root of MSE gives RMSE in the original units, but this week's training objective remains MSE.

5. MSE: square each error, then specify what is averaged

An error of +2 and an error of −2 add to 0, but that does not mean both predictions are correct. Squaring gives 4 and 4. Averaging them measures the mean squared discrepancy per item. This is a suitable choice for our numerical regression example, not the only scoring rule for every task.

\hat y_i=wx_i+b,\quad l_i=(\hat y_i-y_i)^2,\quad L=\frac{1}{n}\sum_{i=1}^{n}l_i

i indexes samples, and n is the number of samples included in the average. Σ means add the terms; 1/n means divide by their count. With n=1, MSE is simply that sample's squared error. The choice of squared error depends on the objective; mean versus sum determines how we combine the individual losses.

Scroll horizontally to view all columns.

Course data table
x	Target y	Squared error
1	3	9
2	5	25
3	7	49
4	9	81

The sum is 164, and the mean is L=164/4=41. Report 164 if you use sum reduction, or 41 if you use mean reduction. The gradients must correspond to the same reduction: do not report mean loss while accidentally updating with gradients of the sum.

MSE has squared target units and penalizes unusually large errors strongly. The next question is this: 41 tells us how poor the current predictions are, but not whether w should increase or decrease.

Knowledge check

EX02-A: One sample has prediction 2 and target 5. What is its MSE? If you repeat the same sample four times, what are the mean loss and sum loss?

6. Why do we also need a gradient?

The model now has a large loss, but that number alone does not say:

In which direction should w and b change, and by how much?

The gradient describes local directions and sensitivities. The learning rate and update rule determine how large a step we take.

7. Derivatives: what happens to loss when w changes slightly?

Fix x=2, y=5, and b=0, and allow only w to change. The prediction is 2w, so L(w)=(2w−5)². Here the horizontal axis is parameter w, not input x. We are changing the prediction rule, not choosing a different question.

python

def loss(w):
    return (w * 2 + 0 - 5) ** 2

print(loss(1.0))      # 9
print(loss(1.001))    # 8.988004
print((loss(1.001) - loss(1.0)) / 0.001)  # approximately -11.996

This is a standalone experiment. Increasing w by 0.001 changes loss by 8.988004−9=−0.011996. Divide by the parameter change to estimate the local change in loss per unit change in w: approximately −11.996. The negative sign means a small move to the right decreases loss here.

\frac{\Delta L}{\Delta w}=\frac{-0.011996}{0.001}\approx-11.996,\qquad \frac{dL}{dw}\bigg|_{w=1}=-12

Δ means a change. dL/dw is the local rate of change obtained as the trial step approaches zero; it is called a derivative. It is not the loss itself: loss is 9 here, while the derivative is −12. Nor does it promise that a whole-unit step decreases loss by 12. The approximation is local and requires a sufficiently small change.

Why is the exact derivative −12? Split the calculation into two stages. Increasing w by δ increases prediction wx+b by xδ=2δ. The current error is e=2−5=−3, and the local rate of change of e² with respect to e is 2e=−6. Multiply the two sensitivities: −6×2=−12. Week 4 extends this multiplication along a dependency path to a whole network.

\frac{dL}{dw}=\underbrace{2(\hat y-y)}_{\text{loss sensitivity to error}}\underbrace{x}_{\text{prediction sensitivity to weight}}

If x=0, changing w does not change this prediction, so this sample contributes zero gradient for w. The bias b can still affect the prediction. This does not mean the model has learned the task; this particular path simply provides no learning signal for w. Next we collect the derivatives for multiple parameters into a gradient.

Knowledge check

EX02-B: Change only w to 1.002. First predict the loss change using −12×0.002, then calculate the actual change. Why are they not exactly equal?

8. What is a gradient?

For a single parameter w:

\mathrm{Gradient} = \frac{d\,\mathrm{Loss}}{dw}

It describes how loss changes locally when w changes slightly.

The sign is especially useful:

gradient > 0
A small increase in w increases loss locally
Decrease w by a suitably small amount

gradient < 0
A small increase in w decreases loss locally
Increase w by a suitably small amount

The parameter update rule is therefore:

w_{\mathrm{new}} = w_{\mathrm{old}} - \mathrm{learning}_{\mathrm{rate}} \times \mathrm{gradient}

Why subtract? The gradient points in the direction of steepest local increase in loss under the usual Euclidean measure. We take a step in the opposite direction.

Scroll horizontally to view all columns.

Course data table
Notation	How to read it	Meaning in this course
dL/dw	The derivative of L with respect to w	Local slope when there is one variable
∂L/∂w	The partial derivative of L with respect to w	Hold the other parameters fixed while considering w
∇θL	The gradient of L with respect to parameter vector θ	Collect the partial derivatives for all parameters
η	eta, the learning rate	The scale we choose for this update step

9. Gradient Descent

One way to picture gradient descent:

You are on a hill. You cannot see the whole landscape, but you can measure the slope where you stand.

The aim is to reach a low point.

calculate gradient

↓

move downhill

↓

calculate gradient again

↓

move downhill again

↓

repeat

In machine learning, a low point corresponds to a small loss.

Scroll horizontally to view all columns.

Course data table
Old parameter w=1; learning rate η=0.1	Update	Direction
gradient=3	1−0.1×3=0.7	Decrease w
gradient=−3	1−0.1×(-3)=1.3	Increase w
gradient=0	1−0=1	No first-order direction signal at this point

Subtracting the gradient attempts a step in a locally descending direction. A large step can overshoot the bottom. For more complex losses, gradient descent does not guarantee that every finite step decreases loss or that it finds a global minimum. A zero gradient means zero first-order change; the point may be a minimum, maximum, or another stationary point.

Knowledge check

If w=2, gradient=−4, and η=0.05, what is the next w?

10. Learning Rate

Parameter update:

\mathrm{parameter}_{\mathrm{new}} = \mathrm{parameter}_{\mathrm{old}} - \mathrm{learning}_{\mathrm{rate}} \times \mathrm{gradient}

The learning rate scales the size of each step.

Too large: it may overshoot, oscillate, or even diverge
Too small: progress can be slow

The learning rate is a hyperparameter.

Distinguish:

Parameters: values learned during training, such as w and b
Hyperparameters: settings chosen for training, such as learning_rate, batch_size, and epochs

11. Work through one complete update of w

Use only one training sample:

x = 2

y = 5

First simplify the model to:

\hat{y} = wx

Initially:

w = 1

learning_rate = 0.01

Step 1 — Prediction

\hat{y} = 1 \times 2 = 2

Step 2 — Loss

\mathrm{Loss} = (\hat{y} - y)^2 = (2 - 5)^2 = 9

Step 3 — Gradient

Loss：

\mathrm{Loss} = (\hat{y} - y)^2

Prediction：

\hat{y} = wx

Apply the chain rule:

\frac{d\,\mathrm{Loss}}{dw} = \left(\frac{d\,\mathrm{Loss}}{d\hat{y}}\right) \times \left(\frac{d\hat{y}}{dw}\right)

First factor:

\frac{d\,\mathrm{Loss}}{d\hat{y}} = 2(\hat{y} - y)

Substitute the values:

2(2 - 5) = -6

Second factor:

dŷ/dw = x = 2

Therefore:

\frac{d\,\mathrm{Loss}}{dw} = -6 \times 2 = -12

Gradient = -12。

The negative derivative means a sufficiently small increase in w decreases loss here.

Step 4 — Update

w_{\mathrm{new}} = 1 - 0.01 \times (-12)

Therefore:

w_{\mathrm{new}} = 1.12

New prediction:

\hat{y} = 1.12 \times 2 = 2.24

New loss:

(2.24 - 5)² = 7.6176

Loss was 9; it is now approximately 7.62.

The model's error on this sample has decreased.

12. Why is the chain rule important?

The dependency is:

↓

prediction ŷ

↓

Loss

To determine how w affects loss, work backward through this dependency chain.

This is the central idea behind backpropagation, which we study later.

13. Bias b and partial derivatives

A partial derivative considers one parameter at a time: for ∂L/∂w, hold b and the data fixed; for ∂L/∂b, hold w and the data fixed. The gradient collects these local slopes in the same order as the parameters. Computing derivatives does not itself update any parameter.

The complete model:

\hat{y} = wx + b

Loss now depends on both w and b:

\mathrm{Loss} = f(w,b)

We calculate separately:

\frac{\partial\,\mathrm{Loss}}{\partial w}

and:

\frac{\partial\,\mathrm{Loss}}{\partial b}

A partial derivative studies how one variable affects the result while the others are held fixed. For a single sample:

\frac{\partial\,\mathrm{Loss}}{\partial w} = 2(\hat{y} - y)x

\frac{\partial\,\mathrm{Loss}}{\partial b} = 2(\hat{y} - y)

The collection of these partial derivatives is the gradient.

14. Update w and b together

Suppose:

w = 1

b = 0

gradient_w = -12

gradient_b = -6

learning_rate = 0.01

Update:

w = 1 - 0.01 × (-12) = 1.12

b = 0 - 0.01 × (-6) = 0.06

New model:

\hat{y} = 1.12x + 0.06

This is one small learning step.

Compute dw and db from the same old w and b, then write both new values. Updating w first and recomputing db using the new w and old b is a different update sequence, not the simultaneous gradient-descent step described here. Keeping the forward-pass values consistent is also essential to Week 4's backpropagation.

15. Second derivatives

The first derivative describes:

The rate of change, or slope

The second derivative describes:

How quickly the rate of change itself changes.

An intuitive analogy:

Position

↓derivative

Velocity

↓derivative

Acceleration

In optimization, second derivatives also describe curvature of the loss surface.

For Week 2, this intuition is enough; a detailed study of the Hessian is not required.

16. Least squares versus gradient descent

These are different concepts.

Least squares answers:

What objective do we want to optimize?

We want to minimize the sum of squared errors.

Gradient descent answers:

How can we move iteratively toward a smaller loss?

So:

Least Squares = Objective

Gradient Descent = Optimization method

Simple linear regression can also have a closed-form solution. Neural networks generally use iterative optimization because their objectives are more complex and do not have a comparable general closed-form solution—not merely because they contain millions or billions of parameters.

17. Implement it from scratch in Python

Scroll horizontally to view all columns.

Course data table
x / y, starting from w=b=0	Prediction	Error	Squared error	2 × error × x	2 × error
1 / 3	0	−3	9	−6	−6
2 / 5	0	−5	25	−20	−10
3 / 7	0	−7	49	−42	−14
4 / 9	0	−9	81	−72	−18
Mean over four samples	—	—	41	dw=−35	db=−12

With η=0.01, updating both parameters gives w=0.35 and b=0.12. The new predictions are [0.47,0.82,1.17,1.52], and the new MSE is 28.45315, below 41. This is the first update, not the end of learning; the loop continues from these new parameters.

Without sklearn, PyTorch, or TensorFlow:

python

x_data = [1, 2, 3, 4]
y_data = [3, 5, 7, 9]
w = 0.0
b = 0.0
learning_rate = 0.01

for epoch in range(1000):
    dw = 0.0
    db = 0.0
    loss = 0.0
    n = len(x_data)

    for x, y in zip(x_data, y_data):
        # Forward pass
        prediction = w * x + b
        # Error
        error = prediction - y
        # Squared error
        loss += error**2
        # Gradients
        dw += 2 * error * x
        db += 2 * error

    # MSE / mean gradients
    loss /= n
    dw /= n
    db /= n

    # Gradient descent
    w -= learning_rate * dw
    b -= learning_rate * db

print('w:', w)
print('b:', b)

The final parameters approach:

w ≈2

b ≈1

In other words, the model learns this relationship from the data:

y \approx 2x + 1

Knowledge check

Why does the first step use −140/4=−35 instead of updating w directly with −140?

18. Match each line of code to the mathematics

Model：

python

prediction = w * x + b

Corresponds to:

\hat{y} = wx + b

Error：

python

error = prediction - y

Corresponds to:

\hat{y} - y

Loss：

python

loss += error**2

Corresponds to:

(ŷ - y)²

Gradient for w：

python

dw += 2 * error * x

Corresponds to:

\frac{\partial\,\mathrm{Loss}}{\partial w} = 2(\hat{y} - y)x

Gradient for b：

python

db += 2 * error

Corresponds to:

\frac{\partial\,\mathrm{Loss}}{\partial b} = 2(\hat{y} - y)

Parameter update：

python

w -= learning_rate * dw
b -= learning_rate * db

Corresponds to:

parameter = parameter - learning_rate × gradient

19. Forward pass and backward pass

Forward Pass：

Input

↓

Model

↓

Prediction

↓

Loss

Backward Pass / Backpropagation：

Loss

↓

Gradients

↓

Optimizer / Gradient Descent：

Gradients → Updated Parameters

20. Epochs and batches

An epoch is one complete pass through the training dataset.

For example:

python

for epoch in range(1000):
    # Use the training data once per epoch.
    ...

This repeats a full pass through the training data 1,000 times.

A batch is the group of samples processed together in a forward/loss/backward/update cycle. It can be the entire training dataset (full batch) or a subset (mini-batch, more common in practice).

For example, a million samples can be processed in batches of 32 or 64.

21. How linear regression connects to neural networks

Linear Regression：

\hat{y} = wx + b

The core computation of a neural-network neuron is also:

z = w_1x_1 + w_2x_2 + \cdots + w_nx_n + b

An activation function often follows this computation, especially in hidden layers.

Linear regression is therefore a useful starting point for understanding a neuron.

22. From linear regression to GPT

Linear Regression：

Input

↓

Parameters

↓

Prediction

↓

Loss

↓

Gradient

↓

Update Parameters

Neural Network：

Input

↓

Many layers / weights

↓

Prediction

↓

Loss

↓

Backpropagation

↓

Update weights

GPT：

Tokens

↓

Transformer

↓

Next-token Prediction

↓

Loss

↓

Backpropagation

↓

Update huge numbers of parameters

The structure and scale grow more complex, but the underlying training sequence follows the same principles.

23. Essential Week 2 vocabulary

English

Plain-language meaning

Core idea

Feature

Input x

Target / Label

Target / label

The known answer y

Model

wx+b

Parameter

w、b

Prediction

Error

ŷ-y

Loss

How the current predictions score under the chosen objective

MSE

Mean squared error

Average squared error over the specified samples, including the case of one sample

Derivative

Slope / rate of change

Partial Derivative

Partial derivative

Local loss sensitivity to one parameter, holding others fixed

Gradient

The collection of parameter-wise partial derivatives

Gradient Descent

Gradient descent

Take a step opposite the gradient to try to reduce loss

Learning Rate

Learning rate

Scale of each update step

Epoch

One full pass through the dataset

Batch

A group of samples processed together

Scroll horizontally to view all columns.

Core Week 2 vocabulary
English	Plain-language meaning	Core idea
Forward Pass	Forward pass	Input → Prediction → Loss
Backward Pass	Backward pass	Loss → Gradients
Chain Rule	Chain rule	Combine local derivatives along dependencies

24. The five most important Week 2 formulas

1. Prediction

\hat{y} = wx + b

2. MSE

\mathrm{MSE} = \frac{1}{n} \times \sum_i (\hat{y}_i-y_i)^2

3. Gradient for w

\frac{\partial\,\mathrm{MSE}}{\partial w} = \frac{2}{n}\sum_i(\hat{y}_i-y_i)x_i

4. Gradient for b

\frac{\partial\,\mathrm{MSE}}{\partial b} = \frac{2}{n}\sum_i(\hat{y}_i-y_i)

5. Gradient Descent

\mathrm{Parameter}_{\mathrm{new}} = \mathrm{Parameter}_{\mathrm{old}} - \mathrm{LearningRate} \times \mathrm{Gradient}

The fifth connects the calculated gradient to an actual parameter update.

25. The final mental model

Remember this sequence:

DATA

↓

PARAMETERS

↓

MODEL

↓

PREDICTION

↓

COMPARE WITH TARGET

↓

LOSS

↓

GRADIENT

↓

UPDATE PARAMETERS

↓

BETTER MODEL

One-sentence summary of the learning process studied here:

Learning = finding parameters that minimize loss.

Programmer's version:

python

while training:
    prediction = model(input, parameters)
    loss = calculate_loss(prediction, actual)
    gradients = calculate_gradients(loss, parameters)
    parameters -= learning_rate * gradients

This is the foundation for Week 3's neural networks, backpropagation, and the later Transformer / GPT model.

Week 2 deeper connections: turn formula results into runnable intuition

This supplement revisits the connections that are easiest to miss. The chapter introduced prediction, loss, derivatives, gradients, and gradient descent; here we place them back into one dependency chain.

1. Loss is not the gradient

Loss is a number scoring the model's predictions under the current parameters. For example:

\mathrm{Loss}(w) = (2w - 5)^2

When w=1, prediction is 2 and loss is 9.

The gradient is the rate of change of loss with respect to the parameters:

\frac{d\,\mathrm{Loss}}{dw} = -12

It answers a different question: if only w increases slightly, does loss increase or decrease, and how quickly?

Loss → the score at the current parameter values

Gradient → local sensitivity to parameter changes

A derivative needs a variable: “the derivative of loss” is incomplete without saying with respect to what. In training, we usually differentiate loss with respect to trainable parameters such as weights and biases.

2. One parameter versus multiple parameters

If w is the only parameter:

\mathrm{gradient} = \frac{d\,\mathrm{Loss}}{dw}

This is one number: the local slope of a one-dimensional curve.

If both w and b are parameters:

\nabla \mathrm{Loss} = \begin{bmatrix} \frac{\partial\,\mathrm{Loss}}{\partial w} \\ \frac{\partial\,\mathrm{Loss}}{\partial b} \end{bmatrix}

The gradient is a vector. Each component gives the local slope along one parameter direction. Together, they point toward the steepest local increase in loss under the usual Euclidean measure. Gradient descent uses the opposite direction:

\theta_{\mathrm{new}} = \theta_{\mathrm{old}} - \alpha \nabla \mathrm{Loss}

3. Why a negative gradient increases the parameter

Suppose:

w = 1

gradient = -12

learning_rate = 0.01

Update:

w_{\mathrm{new}} = 1 - 0.01(-12) = 1.12

The subtraction implements gradient descent's opposite-slope rule. A negative derivative means a small increase in w decreases loss locally. Subtracting that negative number produces a positive update.

4. How the chain rule connects to backpropagation

\frac{d\,\mathrm{Loss}}{dw} = \frac{d\,\mathrm{Loss}}{d\hat{y}}\frac{d\hat{y}}{dz}\frac{dz}{dw}

Each operation can supply its own local derivative, so a node does not need to understand the whole network. Where dependencies branch, contributions must also be summed; Week 4 explains that case.

5. Why second derivatives are not required this week

The first derivative gives the slope at the current point. Our initial training question is which direction to move. That is enough to perform:

\mathit{parameter} \leftarrow \mathit{parameter} - \mathit{learningRate} \times \mathit{gradient}

The second derivative:

\frac{d^2\,\mathrm{Loss}}{dw^2}

It tells us how the slope changes—that is, the curvature. Some optimization methods use this information to help choose steps, but a full Hessian is expensive for networks with millions of parameters. GPT training commonly uses first-order methods such as SGD, momentum-based updates, and AdamW, rather than explicitly constructing a full Hessian.

6. Check the derivative direction with a finite change

A small trial step can approximately check the derivative:

\frac{d\,\mathrm{Loss}}{dw} \approx \frac{\mathrm{Loss}(w+\varepsilon)-\mathrm{Loss}(w)}{\varepsilon}

For w=1 and ε=0.001:

Loss(1)

= 9.000000

Loss(1.001) ≈8.988004

Change

≈-0.011996

Divide by 0.001

≈-11.996

This is close to the analytic result, −12. Finite differences are useful for checking hand-written gradients on small examples, but they are costly as a replacement for backpropagation: checking each parameter requires additional forward evaluations.

7. Close the loop in one sequence

Parameters determine predictions

Predictions and targets determine loss

The gradient of loss with respect to parameters provides local direction information

Gradient descent updates parameters in the opposite direction

With appropriate data, an update rule, and step sizes, repeated updates can improve predictions