Week 3
Week 3 — Neural networks: from one neuron to multiple layers
Key questionHow do we extend one wx+b calculation into a multilayer network?
Learning objectives
- Understand and apply Week 3 — Neural Networks: from One Neuron to Multiple Layers
190 min estimated reading time
Last week, you learned to train a straight line. This week, keep the learning principle and ask a new question: if the target pattern is not a straight line, how should the prediction function change? Work through four units: one neuron → nonlinearity → a two-layer network → code and task-specific outputs.
Recall y_hat=wx+b: the input is the value supplied for this example, while w and b are the rule that training changes. A neural network connects multiple multiply-and-add units like this. The key new ingredient is not the terminology but the nonlinearity between layers. First run ReLU, then examine XOR, and finally calculate and implement a 2→2→1 network.
Scroll horizontally to view all columns.
| This week's experiments | Fixed conditions | What to observe |
|---|---|---|
| Two-layer forward pass | Input [1,2]; two hidden units; a linear output | Hidden values [1.4,1.1], prediction 1.3, and loss 0.49 when the target is 2. |
| Negative-output comparison | Change only the output weights to [−1,0] and bias to 0 | The prediction should be −1.4; a generic layer function must not silently clip it to 0. |
| XOR | Two manually specified ReLU branches | All four inputs can be calculated by hand. This demonstrates representational capacity, not a successful training run. |
Run course_examples/week03_neuron.py; the separate XOR experiment is course_examples/week03_xor.py. By the end, explain parameter counts, hidden-layer activations, and the choice of output layer. You do not yet need to derive every gradient from scratch. Week 4 tackles how each parameter affects the final error.
Unit 1: from linear prediction to one neuron
1. First, why do we need neural networks?
In Week 2, we studied linear regression:
For example, predicting house prices:
Suppose:
w = 5000
b = 100000
Then, for a 100 m² house:
We obtain:
This is a simple relationship:
Floor area increases
↓
Price increases at an approximately constant rate
This is an approximately linear relationship.
But real-world patterns are often more complicated.
2. Real-world patterns often do not form a straight line
Predicting house prices might involve:
Floor area
Number of bedrooms
Age of the property
Distance to the central business district (CBD)
School catchment
Local crime rate
Land area
Renovation condition
Interest rates
Market conditions
...
These factors may also interact.
For example:
A very large house
+
Far from the CBD
It might still command a relatively low price.
But:
A large house
+
A sought-after school catchment
+
Close to the CBD
The combination might command a much higher price.
In other words:
Many real-world patterns are nonlinear.
Neural networks offer one way to model such patterns.
3. Start from Week 2's linear regression
Week 2：
Now suppose there are three features:
- x₁: floor area
- x₂: number of bedrooms
- x₃: distance to the CBD
We can write the model as:
It looks a little more complicated.
But the basic operation is unchanged.
Previously:
Now we have:
That is:
Each input has its own weight.
4. What does a weight mean?
Weights are central to neural networks.
Suppose:
If:
x1 = floor area
x2 = number of bedrooms
x3 = distance to the CBD
Training might produce:
w1 = 0.8
w2 = 0.4
w3 = -0.7
Within this illustrative model, we can read the signs as follows:
Floor area → a positive contribution
Number of bedrooms → a positive contribution
Distance to the CBD → a negative contribution
For this simple linear model, think of a weight as:
The change in the prediction per unit increase in that input, holding the other inputs fixed. Its sign gives the direction. Its magnitude depends on feature units and scale, so weights alone do not provide a universal ranking of importance or establish causation.
In a large neural network, an individual weight usually has no such direct human-readable interpretation.
5. This is already the core of a neuron
The following computation:
is a basic building block of a neural network.
We can draw it as:
x1 ──w1 ──┐
│
x2 ──w2 ──┼── SUM ── z
│
x3 ──w3 ──┘
+
b
This is the simplest form of an:
Artificial Neuron
Artificial neuron.
6. Why call it a neuron?
The idea was inspired by biological neurons, but the two should not be treated as equivalent.
This artificial neuron's computation is straightforward:
Receive inputs
↓
Multiply by weights
↓
Add the products
↓
Add a bias
↓
Produce a number
In mathematical notation:
The symbol:
is a compact way to say “add all the terms.”
For example:
It means:
Σ is simply shorthand for a sum.
7. Calculate one neuron by hand
Suppose we create an illustrative score for whether a house is worth considering.
Three inputs:
x1 = 0.8, a location feature
x2 = 0.6, a property-condition feature
x3 = 0.2, a distance-related feature
Weights：
w1 = 0.7
w2 = 0.5
w3 = -0.4
Bias：
b = 0.1
Calculate:
Substitute:
Step by step:
Therefore:
So far, this is just multiplication and addition.
It means:
z = x1*w1 + x2*w2 + x3*w3 + b
8. But there is an important limitation
If a neuron only computes:
then even if we stack many layers:
Input
↓
Linear
↓
Linear
↓
Linear
↓
Output
the overall input–output function is still affine—a linear model with a bias.
This is a key point in understanding neural networks.
9. Why are stacked linear layers still linear?
Suppose the first layer is:
The second layer is:
Substitute the first layer:
Expand:
Treat:
as a new weight:
Treat:
as a new bias:
Finally:
We find that:
Two consecutive affine layers can be combined into one affine layer.
The same holds for 10 such layers.
And for 100 such layers.
Stacking wx+b operations alone does not add nonlinear representational capacity.
We need another ingredient.
The precise statement is about the input–output function: consecutive linear or affine layers can be collapsed into one, so they do not add nonlinear representational capacity. Splitting them into layers can change the parameterization and optimization dynamics, but depth alone cannot produce a nonlinear decision boundary in this case.
This example has one input and one output. Its combined W and B are scalars, not the later weight matrix W or batch size B. Scalar multiplication allows Wx=xW. With multi-feature matrices, multiplication order must match the shapes.
Unit 2: observe ReLU, then understand nonlinearity
10. Activation Function
That ingredient is:
Activation Function
An activation function.
A neuron no longer computes only:
Instead:
The full calculation:
Here:
- z: the result of the affine multiply-and-add calculation
- f：Activation Function
- a：activation/output
12. A simple activation: ReLU
A widely used activation in neural networks:
ReLU
Full name:
Rectified Linear Unit
Formula:
It may look like abstract notation.
But translated into code, it is:
def relu(x):
return max(0, x)The operation itself is this small.
13. What does ReLU do?
If:
x = 5
Then:
If:
x = -3
Then:
If:
x = 0.7
Then:
Therefore:
negative → 0
positive → keep it
Sketch of its graph:
y
│
│
/
│
/
│
/
│
/
│___/____________ x
0
11. Why do activation functions matter?
Because a nonlinear activation introduces:
Non-linearity
Nonlinearity.
The model is no longer restricted to a single affine input–output relationship.
For example, a relationship might have this shape:
*
* *
*
*
*
*
*
*
Or:
________
\
\
\_______
Or a much more complicated boundary.
Nonlinear components are necessary for a neural network to represent these nonlinear relationships.
14. A complete neuron with an activation
Before:
Inputs
↓
Weights
↓
Sum
↓
Bias
↓
Output
Now:
Inputs
↓
Weights
↓
Weighted Sum
↓
Bias
↓
z = wx+b
↓
Activation Function
↓
a = ReLU(z)
↓
Output
Mathematically:
15. Calculate a neuron with ReLU
Suppose:
Weights：
Bias：
First calculate:
Therefore:
Then apply the activation:
We obtain:
This neuron produces zero for the current input.
16. What if we change a weight?
For example, replace:
with:
Recalculate:
ReLU：
Therefore:
This illustrates an important point:
Weights determine how the neuron responds to inputs.
Training adjusts those weights using data and the chosen loss.
The learning principle is the same as in Week 2.
17. What can one neuron do?
A single neuron still has limited representational capacity.
Greater flexibility comes from:
Multiple neurons working together.
For example:
x1 ─┬── Neuron 1
│
├── Neuron 2
│
└── Neuron 3
x2 ─┬── Neuron 1
│
├── Neuron 2
│
└── Neuron 3
Such a group of neurons forms a:
Layer
Layer
Unit 3: calculate a complete two-layer network
18. What is a layer?
Suppose the input is:
The hidden layer has three neurons:
┌─Neuron 1
x1 ─────┼─Neuron 2
└─Neuron 3
▲
x2 ───────┘
Each neuron has its own weights.
Neuron 1：
Neuron 2：
Neuron 3：
Each result then passes through ReLU:
The layer therefore outputs:
Here, the Python weights store one output neuron's input weights in each row, giving [outputs,inputs]. Week 1's XW+b stores the same numbers with each output in a column of W, giving [inputs,outputs]. The later nn.Linear example uses the first storage convention too. The two weight arrays are transposes of each other.
19. Why put multiple neurons in a layer?
Different neurons can learn different responses or patterns.
To build intuition, consider image recognition.
Some neurons might become responsive to:
Vertical lines
in a particular arrangement.
Others might respond to:
Horizontal lines
Others might respond to:
Edges
Later layers may combine earlier responses into patterns associated with:
Eyes
Ears
Outlines
Further combinations might be associated with:
A cat's face
This is not a programmer explicitly writing:
if has_whiskers and has_ears:
return "cat"Rather, training adjusts weights to reduce the chosen loss. This feature hierarchy is an illustrative possibility, not a guaranteed interpretation of every trained neuron.
21. Feature Learning
Learning intermediate representations is an important feature of neural networks.
For example, suppose the input is an image of a cat.
The raw data may simply be:
pixel
pixel
pixel
pixel
...
The model transforms these values layer by layer:
Pixels
↓
Edges
↓
Shapes
↓
Parts
↓
Object concepts
↓
Cat
This introduces an important AI concept:
“This neuron might detect an ear” is an analogy for combining features, not a promise that every trained unit has a clear human-readable label. Representations are often distributed across many units. A single weight or activation is not enough to prove that the model has learned a real-world concept.
Representation
The model transforms data into representations that can be more useful for the task.
When you later study:
- Embedding
- Attention
- Transformer
you will encounter this idea repeatedly.
22. What does “deep” mean in deep learning?
Suppose:
Input
↓
Hidden
↓
Output
A small number of layers.
If:
Input
↓
Layer 1
↓
Layer 2
↓
Layer 3
↓
Layer 4
↓
Layer 5
↓
...
↓
Output
Many layers.
This is called:
Deep Neural Network
Therefore:
In deep learning, “deep” mainly refers to having multiple layers of learned transformations.
It does not mean “the AI is thinking deeply.”
23. Calculate a small neural network by hand
We will build a very small network:
x1 ───── h1 ───┐
\
│
\
├── output
/
│
/
│
x2 ───── h2 ────┘
Two inputs:
Two neurons in the hidden layer.
26. Notice what changed
Original input:
After the hidden layer:
In other words:
Original representation
[1, 2]
↓Neural Network
New representation
[1.4, 1.1]
This is the idea of:
Representation Transformation
Transformers also repeatedly transform intermediate representations.
27. Output Layer
The output neuron now receives:
h1 = 1.4
h2 = 1.1
Suppose:
Bias：
Calculate:
Substitute:
This completes one:
Knowledge check
If the target can be −2, what limitation would ReLU impose on the final layer?
Forward Pass
Forward Pass
28. The complete forward pass
What we just did was:
x1 = 1
x2 = 2
│
▼
┌──────────────┐
│Hidden Layer │
│
│
│h1 = 1.4
│
│h2 = 1.1
│
└──────────────┘
│
▼
┌──────────────┐
│Output Layer │
│
│
│y' = 1.30
│
└──────────────┘
That ingredient is:
- Input
- Calculate one layer after another
- Prediction
29. We have a prediction—what comes next?
Now reconnect this to Week 2.
Suppose the correct answer is:
The model predicts:
Error：
Using squared error:
What comes next?
The same learning sequence as in Week 2:
Prediction
↓
Loss
↓
Gradient
↓
Update Weights
Except that the parameters are no longer just:
w
b
Instead:
w11
w12
w21
w22
b1
b2
v1
v2
b3
...
30. This is neural-network training
The complete process:
INPUT
│
▼
Hidden Layer
│
▼
Hidden Layer
│
▼
Output
│
▼
Prediction
│
▼
Loss
│
▼
Gradients
│
▼
Update ALL Weights
│
└──────────────┐
│
▼
Next iteration
The central principle from Week 2 has not changed:
31. How do we get gradients for so many weights?
Week 2：
With only:
w
b
differentiation is relatively manageable.
But in a neural network:
x
↓
w1
↓
Neuron
↓
w2
↓
Neuron
↓
w3
↓
Output
↓
Loss
we need to calculate:
Potentially for millions or billions of parameters.
How can we do that?
The answer is:
Backpropagation
This is the focus of Week 4.
32. Before backpropagation, understand the computational graph
Suppose:
First:
Then:
Code:
z = x * w
y = z + 1We can draw it as:
x ──┐
├──× ── z ── +1 ── y
w ──┘
This is called a:
Computational Graph
Computational graph.
A neural network can be represented as a large computational graph of dependent operations.
33. Why is the computational graph useful?
During training:
Forward
Following dependencies forward:
Input
↓
Calculation
↓
Calculation
↓
Prediction
↓
Loss
Backward
Working backward from loss:
Loss
↑
gradient
↑
gradient
↑
gradient
↑
weights
Using what we already saw in Week 2:
Chain Rule
we calculate gradients through the graph.
That ingredient is:
Backpropagation applies the chain rule backward through a computational graph, propagating and combining gradient contributions.
In Week 4, we will calculate this by hand.
34. Why introduce matrices?
So far, we have written:
But what if there are:
1000 inputs
1000 neurons
Would we write:
neuron1 = x1*w11 + x2*w12 + ...
neuron2 = x1*w21 + x2*w22 + ...
...
No—we can organize the values.
Write the inputs as:
Write the weights as:
The whole layer can then be calculated as:
Then apply the activation:
This is the kind of operation you will see in PyTorch.
35. Matrices organize the same arithmetic
For example:
Weights：
The operation simply takes what we just calculated:
Neuron 1:
1×0.5 + 2×0.4
Neuron 2:
1×(-0.3) + 2×0.8
and computes many such results together.
Matrix multiplication is not an extra complication invented for neural networks.
It gives us a way to:
Compute many neurons in one operation.
36. Matrices from a programmer's perspective
For now, you can read:
as an efficiently implemented version of:
for neuron in neurons:
result = 0
for input_value, weight in zip(inputs, neuron.weights):
result += input_value * weightGPUs are well suited to large, parallel matrix multiplications.
This is one reason:
GPUs are important in AI.
Unit 4: code, output choices, and independent practice
37. Implement one neuron in plain Python
Without PyTorch:
def relu(x):
return max(0, x)
def neuron(inputs, weights, bias, activation=relu):
if len(inputs) != len(weights):
raise ValueError("Each input must have one corresponding weight")
total = sum(x * w for x, w in zip(inputs, weights)) + bias
return total if activation is None else activation(total)
inputs = [1, 2]
weights = [0.5, 0.4]
bias = 0.1
output = neuron(inputs, weights, bias)
print(output)We obtain:
1.4000000000000001
This matches the earlier hand calculation.
activation=relu uses ReLU when the argument is omitted; activation=None returns the affine result directly. This function argument selects the computation—it is not a learned parameter like weights or bias. The printed 1.4000000000000001 is a floating-point approximation; 1.4 is sufficient for the hand calculation.
38. Implement a layer
The complete definitions below can be copied together. Run the subsequent hidden-layer example underneath them. Every output neuron receives the same inputs but has its own weight row and bias.
Now use several neurons:
def relu(x):
return max(0, x)
def neuron(inputs, weights, bias, activation=relu):
if len(inputs) != len(weights):
raise ValueError("Each input must have one corresponding weight")
total = sum(x * w for x, w in zip(inputs, weights)) + bias
return total if activation is None else activation(total)
def layer(inputs, weights, biases, activation=relu):
if len(weights) != len(biases):
raise ValueError("Each output neuron must have one bias")
return [neuron(inputs, row, bias, activation)
for row, bias in zip(weights, biases)]Define two neurons:
inputs = [1, 2]
weights = [
[0.5, 0.4],
[-0.3, 0.8],
]
biases = [0.1, -0.2]
hidden = layer(inputs, weights, biases)
print(hidden)We obtain:
[1.4000000000000001, 1.1]
This reproduces our earlier hand calculation.
39. Connect the output layer
# Continue after Section 38; hidden is approximately [1.4, 1.1].
output_weights = [[0.7, 0.2]]
output_biases = [0.1]
prediction = layer(hidden, output_weights, output_biases, activation=None)
print(prediction) # approximately [1.3]
# Counterexample: this regression output must allow negative values.
negative_prediction = layer(hidden, [[-1.0, 0.0]], [0.0], activation=None)
print(negative_prediction) # approximately [-1.4], not [0]Conceptually:
[1, 2]
↓
Hidden Layer
↓
[1.4, 1.1]
↓
Output Layer
↓
Prediction
We have implemented a very small:
The first parameters give 1.4×0.7+1.1×0.2+0.1=1.3. Because this is positive, mistakenly adding ReLU would not expose the bug. The second set deliberately gives −1.4, revealing whether the output is incorrectly clipped to 0. See course_examples/week03_neuron.py for the complete version.
Neural Network
It does not yet perform training.
Week 4 supplies the gradient calculations needed for training.
40. What does a neural network actually learn?
This distinction is essential.
We might say:
The AI learned to recognize a cat.
This can be a useful high-level description of its behavior.
But what actually changes underneath?
The quantities changed during this training are:
and:
That is, many:
Parameters
For example:
0.12531
-0.88211
0.00341
1.27342
...
A model can contain billions of these numbers.
When we say:
The model has learned knowledge
at the implementation level, much of what we mean is:
Training has adjusted a collection of parameter values that determine the model's behavior.
41. Why can more parameters increase capacity?
Linear Regression：
2 parameters
w
b
A small neural network might have:
Hundreds
Or thousands of parameters
Large deep-learning models:
Millions
Billions
Within a suitable architecture, more parameters can provide additional capacity to represent complex relationships.
But remember:
More parameters do not automatically mean better performance or greater intelligence.
Results also depend on:
- Sufficient, suitable data
- An appropriate architecture
- Effective training
- optimization
- regularization
- compute
and other choices.
42. How do we count parameters?
This is a useful practical calculation.
Suppose a layer has:
Input size = 3
Neuron count = 4
For each neuron:
3 weights
1 bias
Therefore:
4 parameters per neuron
In total:
4 neurons × 4
= 16 parameters
Formula:
That is:
The +4 accounts for four biases.
43. Another parameter-count example
Suppose:
Input features = 100
Hidden neurons = 256
Weights：
Bias：
Total parameters:
This single layer has more than twenty thousand parameters.
Manually choosing all the weights is not a practical training method at this scale.
We need to:
Loss
↓
Gradient
↓
Optimizer
↓
Automatic update
44. How neural networks relate to linear regression
Compare Week 2 and Week 3 side by side.
Linear Regression
Input
↓
Linear
↓
Prediction
Neural Network
Then apply the activation:
That is:
Input
↓
Linear
↓
Activation
↓
Linear
↓
Prediction
With more structure:
Input
↓
Linear
↓
ReLU
↓
Linear
↓
ReLU
↓
Linear
↓
Output
No new magic operation has appeared.
The network combines many:
and inserts nonlinear:
Activation
45. Write a layer as a formula: each symbol matches the loop
Scroll horizontally to view all columns.
| Symbol | shape | What it does in code |
|---|---|---|
| X | [B,in] | B samples; each row contains inputs in a fixed feature order. |
| W | [in,out] | Column j assigns the input weights for output j. |
| b | [out] | One bias per output, reused across the batch. |
| Z、A | [B,out] | Z is the pre-activation result; A is the result after applying the function. |
Our matrix convention uses one row per sample. PyTorch Linear stores weight as [out,in], so X @ layer.weight.T + layer.bias corresponds to this formula. Transposing translates between storage arrangements; it is not a different learning algorithm.
Multiplication plus a bias is an affine transformation; a linear transformation is the zero-bias special case. Frameworks commonly call the layer Linear even when it includes a bias. A hidden layer can use ReLU, while a regression output can return Z directly. The symbol f does not require every layer to use the same activation.
46. ReLU is not the only activation
You will also encounter:
Sigmoid
Output range:
This range is useful for representing a probability, although the range alone does not guarantee a calibrated prediction.
Tanh
Output:
ReLU
GELU
Common in Transformer models.
You will encounter it or related variants in GPT-style architectures.
You do not need to memorize the formula yet.
For now, remember:
One central role of an activation function is to introduce nonlinearity.
47. Why does sigmoid have this shape?
Formula:
Focus on its behavior before memorizing its formula.
Compare inputs and outputs:
A very negative x → output close to 0
x = 0 → 0.5
A very positive x → output close to 1
Graph:
1 |
______
|
__/
|
/
.5|--------*
|
/
| __/
0 |___
+---------------- x
This makes it useful for representing:
0 → unlikely
1 → likely
For example, a binary classifier's predicted probability.
48. Classification versus regression
Week 2 focused on:
Scroll horizontally to view all columns.
| Task | Final-layer output | Loss used in this course |
|---|---|---|
| Predict any real-valued quantity | A real value without ReLU clipping | MSE |
| Choose one of V mutually exclusive classes | V raw logits | Cross-entropy, explained in Week 6 |
| Display probabilities or sample a result | Apply Softmax to the logits | This reads the model's output; do not pass the probabilities back into a loss interface that expects logits |
We begin with single-label classification. Multiple labels being true simultaneously is a different task; do not automatically copy the mutually exclusive Softmax rule into that setting.
Regression
Predicting numerical values:
House price = $812,532
Temperature = 27.3
Sales = 1052
Neural networks can also perform:
Classification
For example:
Cat
Dog
Bird
Or:
Spam
Not Spam
Or:
Fraud
Not Fraud
Much of the architecture can remain similar.
One important change is:
The output representation and loss function.
49. The loss function can change too
In Week 2, we used:
MSE
It is a common choice for numerical regression, though not the only one.
For classification, a common choice is:
Cross Entropy Loss
We will explain it in detail later.
For now, establish this connection:
Regression → often MSE
Classification → often Cross Entropy
50. A common misconception: is a neuron an if statement?
Not in the sense of a hand-written semantic rule.
A neuron does not directly contain a rule such as:
if image_has_ears:
cat_score += 1Instead, it calculates:
Training adjusts W and b.
The combined network can then exhibit complex behavior.
It is not the same as a traditional:
Rule-based Programming
51. Traditional Programming vs Machine Learning
In a conventional hand-written program:
Data
+
Rules written by programmer
↓
Output
For example:
if age >= 18:
allow()Machine Learning：
Data
+
Correct Answers
↓
Training
↓
Learn Parameters
↓
Model
Then apply the activation:
New Data
+
Model
↓
Prediction
The important distinction is whether the task-specific rule is explicitly written or fitted through parameters. The implementation can still contain conditionals, such as ReLU's clipping operation.
52. Does GPT also use these building blocks?
Its computation still makes extensive use of:
The Transformer architecture adds more structure.
For example:
Token
↓
Embedding
↓
Attention
↓
Linear
↓
Activation
↓
Linear
↓
...
But it still contains many:
Matrix multiplication
Weights
Bias
Activation
You are not merely studying an obsolete small neural network.
You are learning:
Basic mathematical building blocks used inside GPT.
53. From regression to GPT: what stays, and what changes?
Scroll horizontally to view all columns.
| Stage | What stays | The new challenge |
|---|---|---|
| Linear regression | Inputs, parameters, predictions, loss, and updates | One affine rule can represent only a limited set of relationships. |
| Neural network | Loss guides parameter updates | Hidden layers and nonlinearity provide more flexible representations. |
| Language model | The same backpropagation and parameter-update principles | The output scores vocabulary candidates rather than predicting one continuous quantity. |
| GPT | A model built from many XW+b operations and other differentiable computations | Causal attention lets a position read its visible context: itself and earlier positions. |
This week, you only need to explain the first two rows. Language modeling is not a wholly separate world from regression: the prediction function grows and its inputs and outputs change, but training still follows how parameter changes affect loss. Later weeks add or replace components step by step.
54. Close the book and explain these seven relationships
- A neuron first calculates a weighted sum plus a bias; several neurons can receive the same input.
- With row-wise samples, a layer calculates Z=XW+b and then A=f(Z).
- Consecutive affine layers can still be collapsed into one affine layer, so stacking them alone cannot represent XOR. This does not mean changing the parameterization has no effect on optimization.
- Nonlinearity lets different input regions produce different response behavior; ReLU is max(0,z).
- Human-readable labels for hidden units are analogies, not guaranteed meanings of specific trained coordinates.
- The output layer depends on the task: regression may require negative values; classification scores and losses arrive in Week 6.
- A forward pass only computes results. Gradient calculation and an optimizer update are separate parts of learning. More parameters do not guarantee better generalization.
55. Three essential Week 3 formulas
➀Linear Calculation
Meaning:
Input
×
Weights
+
Bias
➁Activation
For example:
③ A neuron with its activation
Combined:
If you can translate this formula into the following operations:
z = weighted_sum(inputs, weights) + bias
a = activation(z)
you have grasped much of Week 3's core computation.
56. Independent exercises: predict before revealing the answers
EX03-A: Use x=[2,3], w=[0.5,−0.2], and b=0.1. Calculate z, then a=ReLU(z). EX03-B: Change only the second weight to −1 and recalculate both. EX03-C: If this is a regression output that must allow negative predictions, should the second case return z or ReLU(z)?
def neuron(x, w, b, activation):
z = sum(v * weight for v, weight in zip(x, w)) + b
# Exercise fragment: complete the return behavior for activation.
# None means a linear output; "relu" means clip negative values to zero.Recall exercise EX03-D: Three inputs feed four neurons, each with a bias. How many parameters are there? Do the three input values count as model parameters?
Knowledge check
Write your results and reasoning for A–D before opening the reference answers.
57. Week 3 from a software engineer's perspective
Think of a neuron as:
class Neuron:
def __init__(self):
self.weights = ...
self.bias = ...
def forward(self, inputs):
z = dot(inputs, self.weights) + self.bias
return activation(z)A layer as:
class Layer:
neurons = [
Neuron(),
Neuron(),
Neuron(),
]A neural network as:
class NeuralNetwork:
layers = [
Layer(),
Layer(),
Layer(),
]Forward：
x = input
for layer in layers:
x = layer.forward(x)
prediction = xTraining：
for epoch in range(...):
prediction = model.forward(input)
loss = loss_function(prediction, target)
gradients = backward(loss)
update_parameters(gradients)From this viewpoint:
A neural network is a parameterized computational graph.
Training means:
Adjusting parameters in that graph in an attempt to reduce the final loss.
58. Week 3 → Week 4: the key unresolved question
One major question remains.
Our network:
x1 ──w1──□
Hidden Neuron
x2 ──w2──□
│
│w3
▼
Output Neuron
│
▼
Prediction
│
▼
Loss
If the final result is:
Loss = 10
we know the prediction is wrong.
But:
How much should w1 change?
Should w2 increase or decrease?
How sensitive is the final loss to w3?
In a network with 100 layers, how can we compute the effect of a weight near the beginning?
This is the focus of Week 4:
Backpropagation
It connects Week 2's derivatives, partial derivatives, chain rule, and gradient descent.
You will see how:
Week 2 introduced learning with a small number of parameters. Week 3 organized many parameters into a neural network. Week 4 explains how to calculate their gradients consistently so that the whole network can be trained.
Week 3 deeper connections: why nonlinear layers can represent more patterns
This supplement builds on the chapter's detailed explanations and addresses several important “why?” questions.
1. A neuron's inputs, parameters, and outputs
A neuron can be written as:
Here:
- xᵢ is a feature supplied by the current example;
- wᵢ is a trainable multiplier for that feature;
- b shifts the affine score and, for ReLU, the location of the z=0 activation boundary;
- z is the raw score before activation;
- activation(z) is the neuron's response to that score; a nonlinear activation makes the response nonlinear.
A neuron combines several input values into one trainable response. With many neurons side by side, each can learn a different combination of inputs.
2. Why follow a weighted sum with an activation?
If the entire network contains only linear or affine operations:
the input–output function can still be collapsed into one affine operation. More layers do not, by themselves, extend it beyond that function class.
A nonlinear activation breaks this general collapse into a single affine transformation. ReLU retains positive scores and clips negative ones to zero; sigmoid maps a real score into (0,1), a range useful for a probability prediction when paired with an appropriate task and loss.
4. Why is XOR a classic example?
XOR outputs 0 for (0,0) and (1,1), and 1 for (0,1) and (1,0). One affine score with a straight-line decision boundary cannot separate these two groups.
Two suitably chosen ReLU hidden units can respond on different sides of their activation boundaries; an output unit then combines their values, as in our four-row example. The point is not that GPT still solves XOR by hand. XOR is a small demonstration that nonlinearity and composition can change the functions a network can represent.
5. Why do classification outputs differ?
Regression often predicts a numerical value such as a house price. Binary classification commonly produces one logit, which sigmoid converts to a probability. Mutually exclusive multiclass classification produces one logit per class, and Softmax converts them into a probability distribution. A loss interface that expects logits receives the raw scores directly, not those already-converted probabilities.
6. Why does parameter count matter?
If a layer has n inputs and m neurons:
Add m biases. More parameters mean more adjustable values and greater storage requirements. They can increase representational capacity, but learning useful settings requires suitable data and training signals. Extra capacity may reduce underfitting or increase overfitting risk; neither outcome is guaranteed by parameter count alone.
7. Connect the forward pass to learning
The chapter's forward computation produces predictions but does not say how the weights should change. Week 4 works backward through the same network graph to calculate:
Loss
↓local derivatives + Chain Rule
A gradient for every weight and bias
↓optimizer
updated parameters
The statement “this network does not yet learn” matters: architecture defines the computation and its representational possibilities; backpropagation calculates gradients, and a separate optimizer uses them to update parameters from data.