Week 12

Week 12 — End-to-end Mini GPT: from text and training to generation after loading

Key questionHow do we trace one dataset through tokenizer, MiniGPT, loss, gradients, AdamW, checkpoints and target-free generation, identifying each state change?

Learning objectives

Trace the complete forward: [3,3]→[3,2]→[3,2,4]→two blocks→[3,2,5]→[6,5]+[6]→loss[].
Trace gradients back from scalar loss; distinguish gradient storage, parameter values and AdamW state.
Complete 100 successful updates, an eval/no-grad reference, a canonical checkpoint, strict restoration and generation using the restored model.
Use pipeline, fixed-batch learning and held-out generalization as separate diagnostic stages, connecting Weeks 1–12 into one mental model.

130 min estimated reading time

Week progress: 0 of 25 sections (0%)Course progress: 0 of 321 sections (0%)

The final week is not another list of components to memorize. Complete three small deliverable tasks using the same Week 10 MiniGPT class. The mechanism demonstration and independent-document experiment have explicitly different data, configurations and checkpoint formats.

Scroll horizontally to view all columns.

Course data table
Learning unit	Problem to solve
Task A: Trace	Run the five-token demonstration and record an input, target, representation, logits, loss, gradient, update and newly generated token.
Task B: Independent documents	Run week12_generalization.py to train and validate on the bundled original short documents. Save actual CSV logs, curves and generated samples.
Task C: Control one variable	Write a prediction first, then change just learning rate, head count or layer count while keeping data and evaluation fixed.
Final explanation	Distinguish a correct pipeline, learning the training data and evidence from independent validation.

Every reported result must come from an actual command. The experiment does not ship invented success logs or curves. Record lack of improvement honestly too. By the end, trace a prompt to one new token and explain the limits of tiny data and short context.

Alternate reading, hand calculation and code changes. Each unit can take several sessions. Section numbers remain stable for old links and references; follow the page from top to bottom rather than jumping around to restore numerical order.

Week 12 Goal: One Complete, Traceable Pipeline

MiniGPT computes conditional scores; the training controller turns errors into updates. The fixed English trace uses mini-gpt-v1: [you, like, AI, study, we] at IDs 0..4; B=3, N=3, T=2, C=4, H=2, d_head=2, V=5, n_layer=2. General APIs allow other positive B values, but this training example fixes B=3.

Scroll horizontally to view all columns.

mini-gpt-v1 ordered vocabulary，V=5
ID	token
0	you
1	like
2	AI
3	study
4	we

raw text → raw IDs [B,N]=[3,3]
shift → inputs/targets [B,T]=[3,2]
token + position embeddings [B,T,C]=[3,2,4]
two canonical pre-norm Blocks [3,2,4]
bias-free LM head logits [B,T,V]=[3,2,5]
reshape logits [6,5] + targets [6] → mean loss []
backward → AdamW step → checkpoint → strict restore
restored target-free context → last logits [B,5] → next ID [B,1]

Scroll horizontally to view all columns.

Training scores all six positions; each generation round uses only the current final position.
Supervised task	context	target
1	[you]	like
2	[you,like]	AI
3	[we]	like
4	[we,like]	you
5	[you]	study
6	[you,study]	AI

\mathrm{text}\to\mathrm{IDs}\in\mathbb{N}^{3\times3}\to(x,y)\in\mathbb{N}^{3\times2}\to Z\in\mathbb{R}^{3\times2\times5}\to\mathcal L\in\mathbb{R}

Knowledge check

Where does supervised training first produce a scalar objective?

1. Project Structure: Responsibilities Before Filenames

This week does not introduce another configuration class or a second MiniGPT.

Scroll horizontally to view all columns.

Course data table
module / object	Responsibility	Key interface or long-lived state
mini_gpt_walkthrough.py	Week 10 canonical architecture	GPTConfig、MiniGPT、stable member names、_init_weights
week11_training_and_generation.py	lifecycle controller	train_mini_gpt_step、validators、restore、stable sampling
week12_end_to_end.py	Assemble the fixed trace	batch、100 updates、reference、save/load、generate
checkpoint file	Persist compatible state	schema、tokenizer identity、config、untied weights、AdamW、completed_updates

\theta=\{E_{\mathrm{token}},E_{\mathrm{position}},\theta_{\mathrm{block\,1}},\theta_{\mathrm{block\,2}},\theta_{\mathrm{final\,norm}},W_{\mathrm{head}}\}

Scroll horizontally to view all columns.

Course data table
Actual file	Observable output
course_examples/mini_gpt_walkthrough.py	520 parameters, input/output shapes and a causality check
course_examples/week11_training_and_generation.py	Importable training, evaluation, resume and generation functions
course_examples/week12_end_to_end.py	100 updates, before/after probabilities, matching checkpoint restoration and generated text
course_examples/verify_learning.py	Numerical/behavioral checks, not a language-quality benchmark

Start with python week12_end_to_end.py in the English course_examples directory. It writes mini-gpt-training.pt in the current directory and uses the three fixed sentences, with no corpus download. See that package's README.md for dependencies, environment creation and exercise order.

Knowledge check

Which owner maps [B,T] to [B,T,V]?

2. First Run: Verify the Pipeline

Print the three raw ID sequences and six questions, then inspect dtype, device, shapes, finite loss, gradients and parameter changes.

week12_end_to_end.py

# excerpt: Week 12 pipeline gate
inputs, targets = make_fixed_batch(device)
assert inputs.dtype == targets.dtype == torch.long
assert inputs.shape == targets.shape == (3, 2)
before = {name: value.detach().clone() for name, value in model.named_parameters()}
loss, grad_norm, completed_updates = train_mini_gpt_step(
    model, optimizer, inputs, targets, device, completed_updates=0
)
assert completed_updates == 1
assert torch.isfinite(loss) and torch.isfinite(grad_norm)
assert any(
    not torch.equal(before[name], value.detach())
    for name, value in model.named_parameters()
)

\mathcal L_{\mathrm{uniform}}=-\log(1/V)=\log 5\approx1.609

Scroll horizontally to view all columns.

Course data table
boundary	Fixed evidence	If this fails, inspect
tokenizer/batch	[3,3]→[3,2]，long，IDs 0..4	Mapping and shift
forward	[3,2]→[3,2,5]，loss [] finite	rank、device、mask、targets
backward	expected .grad non-None/finite	Graph and gradient-clearing order
step	At least one parameter value changes; count=1	Optimizer binding and step

Knowledge check

What should you inspect before asking whether the text is good?

3. Start with Text: Freeze the Tokenizer for Stable Addresses

This teaching tokenizer splits known words on whitespace. It has no specials, PAD or UNK.

Scroll horizontally to view all columns.

Course data table
ID	token
0	you
1	like
2	AI
3	study
4	we

week12_end_to_end.py

# excerpt from the assembled Week 12 caller
RAW_TEXTS = ("you like AI", "we like you", "you study AI")
EXPECTED_RAW_IDS = ((0, 1, 2), (4, 1, 0), (0, 3, 2))
FROZEN_STOI = {
    token: token_id
    for token_id, token in enumerate(CANONICAL_ORDERED_TOKENS)
}

def encode_mini_gpt_v1(text: str) -> list[int]:
    pieces = text.split()
    if not pieces:
        raise ValueError("mini-gpt-v1 text must contain a token")
    try:
        return [FROZEN_STOI[piece] for piece in pieces]
    except KeyError as error:
        raise ValueError(
            f"mini-gpt-v1 has no unknown-token fallback: {error.args[0]}"
        ) from error

encoded_rows = [encode_mini_gpt_v1(text) for text in RAW_TEXTS]
assert tuple(tuple(row) for row in encoded_rows) == EXPECTED_RAW_IDS

\operatorname{encode}(\text{you like AI})=[0,1,2]\in\mathbb N^3,\qquad X_{\mathrm{raw}}\in\mathbb N^{B\times N}=\mathbb N^{3\times3}

The tokenizer decides pieces and IDs. Embeddings and blocks learn continuous representations during training. ID 4 is not a quantity or a meaning greater than ID 0; it is simply the fifth vocabulary row, naming we in this edition.

Knowledge check

Which interfaces must change together when adding a token?

4. Training Batch: One Sentence Becomes Two Supervised Questions

Use the left two columns of raw IDs as inputs and the right two as targets. Teacher forcing supplies real prefixes, not tokens just sampled by the model.

Scroll horizontally to view all columns.

Course data table
row	raw IDs [N=3]	inputs [T=2]	targets [T=2]	Two questions
b=0	[0,1,2]	[0,1]	[1,2]	you→like; [you,like]→AI
b=1	[4,1,0]	[4,1]	[1,0]	we→like; [we,like]→you
b=2	[0,3,2]	[0,3]	[3,2]	you→study; [you,study]→AI

python

raw = torch.tensor(
    [[0, 1, 2], [4, 1, 0], [0, 3, 2]],
    dtype=torch.long,
    device=device,
)  # [B,N] = [3,3]
inputs = raw[:, :-1]   # [[0,1],[4,1],[0,3]] [B,T]=[3,2]
targets = raw[:, 1:]   # [[1,2],[1,0],[3,2]] [B,T]=[3,2]

x=X_{\mathrm{raw}}[:,0:N-1],\qquad y=X_{\mathrm{raw}}[:,1:N],\qquad x,y\in\mathbb N^{3\times2}

Knowledge check

What is the target for study in the third input row?

5. Embedding Lookup: Turn Discrete Addresses into Four Learnable Features

token_embedding selects a token row; position_embedding selects a position row. Their sum enters the block.

python

B, T = inputs.shape
positions = torch.arange(T, device=inputs.device)  # [T]=[2]
token_rows = model.token_embedding(inputs)         # [B,T,C]=[3,2,4]
position_rows = model.position_embedding(positions) # [T,C]=[2,4]
x = token_rows + position_rows                     # [3,2,4]

Scroll horizontally to view all columns.

Course data table
axis	Value	Meaning
B=3	Three rows	Three sequences
T=2	Two columns	Two input positions per sequence
C=4	Four channels	Learned features, not manually named attributes

X^{(0)}=E_{\mathrm{token}}[x]+E_{\mathrm{position}}[0:T]\in\mathbb R^{3\times2\times4}

Knowledge check

Why can [3,2,4] be added to [2,4]?

6. Transformer Block: Read Across Positions, Then Transform Each Position

Two canonical blocks process representations [3,2,4] in sequence. Attention mixes across permitted T positions; the FFN mixes C channels separately at each position.

python

# exact Week 10 member names
def forward(self, x: torch.Tensor) -> torch.Tensor:
    x = x + self.attention(self.ln1(x))       # [3,2,4] + [3,2,4]
    x = x + self.feed_forward(self.ln2(x))    # [3,2,4] + [3,2,4]
    return x                                  # [3,2,4]

X'=X+\operatorname{Attention}(\operatorname{LN}_1(X)),\qquad X^{\mathrm{next}}=X'+\operatorname{FFN}(\operatorname{LN}_2(X'))

Scroll horizontally to view all columns.

Course data table
component	Reading scope	Internal width	external shape
causal Attention	Permitted current/left-side positions	H=2，d_head=2	[3,2,4]→[3,2,4]
FFN	Each position independently	C→4C→C, here 4→16→4	[3,2,4]→[3,2,4]

Knowledge check

Which sublayer lets position 1 read position 0?

7. Trace Q/K/V Shapes: Make Context Queries Concrete Matrices

Query describes what the destination position seeks; Key describes how a source position matches; Value contains what is brought back.

Scroll horizontally to view all columns.

Course data table
operation	shape	axes
x	[3,2,4]	[B,T,C]
qkv(x)	[3,2,12]	[B,T,3C]
q,k,v	each [3,2,4]	[B,T,C]
split/transpose	each [3,2,2,2]	[B,H,T,d_head]
q @ kᵀ	[3,2,2,2]	[B,H,query T,key T]
masked Softmax	[3,2,2,2]	visible key columns sum to 1
weights @ v	[3,2,2,2]	[B,H,T,d_head]
merge + output_projection	[3,2,4]	[B,T,C]

S=QK^{\mathsf T}/\sqrt{d_{\mathrm{head}}}\in\mathbb R^{B\times H\times T\times T},\qquad d_{\mathrm{head}}=C/H=2

A=\operatorname{softmax}(\operatorname{mask}(S)),\qquad \operatorname{Attention}(Q,K,V)=AV

Knowledge check

What is the score-matrix shape for one head of one example?

8. LM Head: Score Five Candidates at Every Position

After two blocks, shape is still [3,2,4]. final_norm preserves it, and lm_head produces five scores at each position.

python

for block in model.blocks:
    x = block(x)                 # [3,2,4]
x = model.final_norm(x)          # [3,2,4]
logits = model.lm_head(x)         # [3,2,5], bias=False

Z=\operatorname{LN}_{\mathrm{final}}(X^{(2)})W_{\mathrm{head}}^{\mathsf T}\in\mathbb R^{3\times2\times5},\qquad W_{\mathrm{head}}\in\mathbb R^{5\times4}

Scroll horizontally to view all columns.

Course data table
selected row	Context represented	Five-candidate order
logits[0,0,:]	[you]	[you,like,AI,study,we]
logits[0,1,:]	[you,like]	[you,like,AI,study,we]

Knowledge check

Why are there exactly five logits per position?

9. Cross Entropy: Combine Six Predictions into One Scalar Loss

Reshape logits [3,2,5] to [6,5] and targets [3,2] to [6]. Keeping the same row order preserves all six alignments.

python

flat_logits = logits.reshape(6, 5)
flat_targets = targets.reshape(6)
loss = torch.nn.functional.cross_entropy(flat_logits, flat_targets)
assert loss.ndim == 0

\mathcal L=-\frac1{BT}\sum_{b=1}^{B}\sum_{t=1}^{T}\log p_\theta(y_{b,t}\mid x_{b,\le t}),\qquad Z_{\mathrm{flat}}\in\mathbb R^{6\times5}

Knowledge check

What shape results from flattening targets [3,2]?

10. Backward Trace: From One Number to Every Participating Parameter

loss.backward() traverses the graph that produced loss and accumulates derivatives in participating parameters' .grad. Parameter values are still unchanged at this point.

loss []
six logit rows [6,5]
lm_head.weight.grad [5,4] + final_norm grads
Block 2 grads → Block 1 grads
token/position embedding table grads [5,4] / [2,4]
New parameter values appear after optimizer.step().

python

optimizer.zero_grad(set_to_none=True)
logits, loss = model(inputs, targets)
assert loss is not None and loss.ndim == 0
loss.backward()
assert model.lm_head.weight.grad is not None
assert model.lm_head.weight.grad.shape == (5, 4)
assert model.token_embedding.weight.grad is not None
assert model.token_embedding.weight.grad.shape == (5, 4)
# Parameters still hold pre-step values here.

g_\theta=\nabla_\theta\mathcal L,\qquad \theta_{\mathrm{new}}=\theta_{\mathrm{old}}\ \text{until optimizer.step() succeeds}

Knowledge check

What changes after backward but before step?

11. Embedding Rows: Lookup Gives Direct Gradients to Selected Rows

Inputs contain IDs {0,1,3,4}; ID 2/AI appears only in targets. token_embedding and lm_head are untied.

Scroll horizontally to view all columns.

Course data table
ID/token	input lookup?	target candidate?	Direct path in this batch
0/you	Yes	Yes	token row lookup + LM-head target/non-target scoring
1/like	Yes	Yes	token row lookup + LM-head scoring
2/AI	No	Yes	No direct lookup gradient for token_embedding row 2; lm_head row 2 participates in target scoring
3/study	Yes	Yes	token row lookup + LM-head scoring
4/we	Yes	No	Token-row lookup; output-head row 4 also participates as a non-target Softmax candidate

\frac{\partial\mathcal L}{\partial E_{\mathrm{token}}[i]}=\sum_{(b,t):x_{b,t}=i}\frac{\partial\mathcal L}{\partial X^{(0)}_{b,t}}

python

row_grad_norms = model.token_embedding.weight.grad.norm(dim=1)
print(row_grad_norms)  # exact values depend on initialization/device
# Inspect finiteness and expected direct-use pattern; do not hard-code magnitudes.

No direct lookup gradient for input-embedding row 2 does not mean AI has no effect on loss. As a target it affects untied output-head gradients and, through logits/loss, the upstream representations that were used.

Scroll horizontally to view all columns.

Course data table
Observation point	Row-2 object	What this trace establishes
After backward	dense .grad[2]	Direct lookup contribution is zero
Before step	AdamW moments + weight_decay policy	Different state from the current .grad
After step	token_embedding.weight[2] parameter value	May move through decay, or through historical moments after resumption

python

# excerpt immediately around one fresh runner update
optimizer.zero_grad(set_to_none=True)
_, loss = model(inputs, targets)
assert loss is not None
row2_before = model.token_embedding.weight[2].detach().clone()
loss.backward()
row2_current_grad = model.token_embedding.weight.grad[2].detach().clone()
assert torch.count_nonzero(row2_current_grad).item() == 0
optimizer.step()  # AdamW uses weight_decay=1e-2 and any restored moments
row2_after = model.token_embedding.weight[2].detach().clone()
print("row2_moved_after_step=", not torch.equal(row2_before, row2_after))

Knowledge check

Which ID appears in targets but not inputs, and where does it have a direct effect?

12. Optimizer Step: Read Gradients, Update Parameters and AdamW State

AdamW retains moments/counters for canonical parameters. Week 11's validator also checks a single group with matching parameter identity/order.

Scroll horizontally to view all columns.

Course data table
operation	parameter.grad	parameter values	AdamW state / count
zero_grad	Cleared	Unchanged	Unchanged
forward + loss	No new value yet	Unchanged	Unchanged
backward	Written/accumulated	Unchanged	Unchanged
clip	Possibly rescaled in place	Unchanged	Unchanged
optimizer.step	Read	Updated	Moments/counters updated
After a successful step	Still present, awaiting clearing	Already updated	completed_updates += 1

m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\qquad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

\widehat m_t=\frac{m_t}{1-\beta_1^t},\qquad \widehat v_t=\frac{v_t}{1-\beta_2^t}

\theta_t=(1-\eta\lambda)\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Plain SGD uses θ_t=θ_{t-1}−ηg_t. AdamW does not directly use raw g_t as its update. Even with a zero current row gradient, (1−ηλ) can change nonzero parameters, and restored m/v history can yield a nonzero adaptive term.

python

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
completed_updates += 1  # only after the successful step returns

Knowledge check

Which operation first changes the model.state_dict() parameter values during the training step?

13. Project A: Train the Three Sentences, Save and Reload

The file below imports Week 10's mini_gpt_walkthrough.py and Week 11's week11_training_and_generation.py. It calls their fixed APIs without redefining GPTConfig, MiniGPT, initialization, state-dict names, checkpoint schema or sampling.

week12_end_to_end.py

# week12_end_to_end.py
# This file is a caller. It reuses, rather than redefines, Week 10/11 APIs.
from __future__ import annotations

from pathlib import Path

import torch
from course_data import DEMO_DOCUMENTS, FIVE_WORD_TOKENIZER, configure_console

from mini_gpt_walkthrough import (
    GPTConfig,
    MiniGPT,
    save_mini_gpt_training_checkpoint,
)
from week11_training_and_generation import (
    CANONICAL_ORDERED_TOKENS,
    CANONICAL_TOKENIZER_POLICY,
    CANONICAL_TOKENIZER_VERSION,
    generate_mini_gpt_sampled,
    load_mini_gpt_training_resume,
    train_mini_gpt_step,
    validate_mini_gpt_adamw_completed_updates,
)


RAW_TEXTS = DEMO_DOCUMENTS
EXPECTED_RAW_IDS = (
    (0, 1, 2),  # you like AI
    (4, 1, 0),  # we like you
    (0, 3, 2),  # you study AI
)
FROZEN_STOI = {
    token: token_id
    for token_id, token in enumerate(CANONICAL_ORDERED_TOKENS)
}
assert CANONICAL_ORDERED_TOKENS == FIVE_WORD_TOKENIZER.tokens
assert CANONICAL_TOKENIZER_POLICY == (
    "whitespace-delimited;no-specials;no-pad;no-unk"
)


def encode_mini_gpt_v1(text: str) -> list[int]:
    ids = FIVE_WORD_TOKENIZER.encode(text)
    if not ids:
        raise ValueError("mini-gpt-v1 text must contain a token")
    return ids


def make_fixed_batch(device: torch.device) -> tuple[torch.Tensor, torch.Tensor]:
    encoded_rows = [encode_mini_gpt_v1(text) for text in RAW_TEXTS]
    assert tuple(tuple(row) for row in encoded_rows) == EXPECTED_RAW_IDS
    raw = torch.tensor(encoded_rows, dtype=torch.long, device=device)  # [3,3]
    inputs = raw[:, :-1]   # [3,2]
    targets = raw[:, 1:]   # [3,2]
    return inputs, targets


def main() -> None:
    configure_console()
    # One CPU thread reduces scheduling overhead for this tiny teaching model.
    torch.set_num_threads(1)
    seed = 7
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = MiniGPT(GPTConfig()).to(device)
    optimizer = torch.optim.AdamW(
        model.parameters(),
        lr=1e-3,
        weight_decay=1e-2,
    )
    inputs, targets = make_fixed_batch(device)
    assert inputs.shape == targets.shape == (3, 2)

    # Observe the labelled [you,like] -> AI row before any update.
    model.eval()
    with torch.no_grad():
        before_logits, _ = model(inputs)
        before_probs = torch.softmax(before_logits[0, 1, :], dim=-1)  # [5]

    initial_parameters = [p.detach().clone() for p in model.parameters()]
    with torch.no_grad():
        _, initial_loss = model(inputs, targets)
    completed_updates = 0
    for _ in range(100):
        loss, grad_norm, completed_updates = train_mini_gpt_step(
            model,
            optimizer,
            inputs,
            targets,
            device,
            completed_updates=completed_updates,
        )
        assert torch.isfinite(loss) and torch.isfinite(grad_norm)
    assert completed_updates == 100
    assert any(not torch.equal(old, new)
               for old, new in zip(initial_parameters, model.parameters()))
    validate_mini_gpt_adamw_completed_updates(
        model,
        optimizer,
        completed_updates,
    )

    model.eval()
    with torch.no_grad():
        reference_logits, reference_loss = model(inputs, targets)
    assert reference_logits.shape == (3, 2, 5)
    assert reference_loss is not None and reference_loss.ndim == 0
    after_probs = torch.softmax(reference_logits[0, 1, :], dim=-1)  # [5]
    print("parameters=", sum(p.numel() for p in model.parameters()))
    print("completed_updates=", completed_updates)
    print("same_batch_loss_before=", initial_loss.item())
    print("same_batch_loss_after=", reference_loss.item())
    print("observed context=[you,like], target=AI")
    print("vocabulary_order=", CANONICAL_ORDERED_TOKENS)
    print("before_probs=", before_probs.detach().cpu().tolist())
    print("after_probs=", after_probs.detach().cpu().tolist())

    path = Path("mini-gpt-training.pt")
    save_mini_gpt_training_checkpoint(
        str(path),
        model=model,
        optimizer=optimizer,
        completed_updates=completed_updates,
        ordered_tokens=CANONICAL_ORDERED_TOKENS,
        tokenizer_policy=CANONICAL_TOKENIZER_POLICY,
        tokenizer_version=CANONICAL_TOKENIZER_VERSION,
    )
    restored, restored_optimizer, restored_updates = (
        load_mini_gpt_training_resume(str(path), device=device)
    )
    assert restored_updates == completed_updates
    validate_mini_gpt_adamw_completed_updates(
        restored,
        restored_optimizer,
        restored_updates,
    )

    restored.eval()
    with torch.no_grad():
        restored_logits, _ = restored(inputs)
    torch.testing.assert_close(restored_logits, reference_logits)
    print("checkpoint_round_trip=PASS")

    prompt = torch.tensor(
        [[FROZEN_STOI["you"], FROZEN_STOI["like"]]],
        dtype=torch.long,
        device=device,
    )
    generated_history = generate_mini_gpt_sampled(
        restored,
        prompt,
        max_new_tokens=3,
        temperature=0.8,
        top_k=3,
    )
    assert generated_history.shape == (1, 5)
    print("generated_ids=", generated_history[0].tolist())
    print("generated_text=", " ".join(
        CANONICAL_ORDERED_TOKENS[i] for i in generated_history[0].tolist()))
    print("This demonstrates mechanisms on three fixed sentences, not held-out generalization.")


if __name__ == "__main__":
    main()

P=5\cdot4+2\cdot4+2(8+48+20+8+80+68)+8+5\cdot4=520

Scroll horizontally to view all columns.

Course data table
checkpoint identity field	Fixed content
schema	mini-gpt-training-checkpoint / version 1
tokenizer	mini-gpt-v1 ordered tokens + exact policy + SHA-256
tokenizer SHA-256	38d630f4c589664c9bef567457d48764cbe2307734777e80f7d5d5c63ac88dd6
config	GPTConfig(5,2,4,2,2), no dropout
weight policy	token_embedding_lm_head=untied
training state	model_state + exact AdamW class/state + completed_updates=100

Loss trajectories and generated tokens depend on initialization, device and software version. The program observes and verifies results instead of the textbook inventing them. As an optional comparison, sharing the 5×4 table gives 500 parameters but changes gradient sharing and checkpoint policy. This trace does not enable tying.

Scroll horizontally to view all columns.

Course data table
runtime observation	label / shape	How to report it
before_probs	[you,like]→AI probabilities [5] before updating	Print in CANONICAL_ORDERED_TOKENS order
after_probs	The same row [5] after 100 successful updates	Compute and print from reference_logits[0,1,:] at runtime
interpretation	An observation for one seed/device/version	Do not hard-code values, assert a direction or generalize it into a guarantee

The runner records before_probs under eval/no_grad, trains, then calculates after_probs from post-update reference_logits. Both are labeled context=[you,like], target=AI and printed with vocabulary order. Readers can inspect what this run did rather than mistake textbook numbers for a promise.

Knowledge check

Why is saved completed_updates 100 rather than 99?

14. Autoregressive Generation: Crop Context, Select the Last Row, Sample One ID

Start from the restored model and prompt [[0,1]]=[you,like]. Supply no targets, so loss is None and no learning update occurs.

uncropped history [1,2] = [[0,1]]
crop forward context to last block_size=2 IDs [1,2]
restored(context), targets=None → logits [1,2,5]
logits[:,-1,:] → last_logits [1,5]
promote/row-center/divide by τ → optional top-k → checked Softmax [1,5]
multinomial → next_id torch.long [1,1]
append to uncropped history → [1,3]，repeat

[B,L_{\mathrm{history}}]\to[B,\min(L_{\mathrm{history}},2)]\to[B,T_{\mathrm{context}},5]\to[B,5]\to[B,1]\to[B,L_{\mathrm{history}}+1]

Scroll horizontally to view all columns.

Course data table
phase	positions consumed	targets?	state change
training	all B×T=6 logits rows	Present, [3,2]	backward + step update θ/AdamW
generation	only final row per prompt [B,5]	Absent	Only caller history grows

Knowledge check

What shape is logits[:,-1,:] when logits is [3,2,5]?

15. Why Naive Generation Recomputes the Past

O(T²) describes growth: one head compares each of T queries with T keys, creating T² scores. Increasing T from 2 to 4 changes four scores into sixteen. It does not mean T² seconds of runtime, nor include all FFN/projection costs.

With history=[0,1,2] and block_size=2, the next forward receives [1,2], but recomputes both positions' activations through both layers.

Scroll horizontally to view all columns.

Course data table
strategy	What is retained?	Work per round	trade-off
Naive loop	caller full token IDs	Recompute all layers for the cropped context	Clearest and easiest to inspect
KV cache	Past keys/values per layer plus position state	Primarily compute the new token	Can be faster, but requires precise cache shapes, devices and reset rules

\mathrm{attention\ score\ work\ per\ full\ prefix}=O(T_{\mathrm{context}}^2),\qquad \sum_{t=1}^{L}O(t^2)=O(L^3)

A KV cache can reuse past keys/values and preserve predictions within numerical tolerance when prefixes, positions and computation semantics remain consistent. It cannot be inserted directly into this example: after sliding-window cropping, positions restart at zero and retained tokens may have different position representations and visible prefixes. Old cached states are not automatically equivalent. This teaching code recomputes fully; caching is a separate extension.

Scroll horizontally to view all columns.

Course data table
Computation setting	Attention-score work per round	Cumulative growth with generated length
No cropping; prefix keeps growing	Approximately t²	Summing gives approximately L³ for score work only
This example sees at most two tokens	At most four scores per head	Fixed per-round bound; cumulative work grows roughly linearly with rounds

Knowledge check

Why is there still repeated computation with block_size=2?

16. Three Diagnostic Stages: Pipeline, Fixed-Batch Learning, Generalization

The stages ask different questions: does the code connect correctly, can it learn expressible patterns in this batch, and do those patterns help on unseen data?

Scroll horizontally to view all columns.

Course data table
stage	What stays fixed?	evidence	stop condition
1 Pipeline correctness	fixed batch/config/seed	[3,2]→[3,2,5]→[]；finite grads；parameter changes	Stop if any interface fails
2 Same-batch learning	Repeat the same fixed batch	Loss falls substantially below ln(5); distinct contexts separate; [you] moves toward empirical 0.5/0.5	Do not demand zero loss or six correct argmax decisions
3 Generalization	A frozen train/held-out split of a larger corpus	Read-only eval/no_grad, token-weighted held-out loss and samples	Only this stage measures generalization

\mathcal L_{\mathrm{validation}}=\frac{\sum_{i\in\mathrm{held\text{-}out\ valid\ targets}}\ell_i}{\sum_{\mathrm{held\text{-}out\ batches}}\#\mathrm{valid\ targets}}

week12_validation_excerpt.py

@torch.no_grad()
def evaluate_token_weighted(model, held_out_batches, device, ignore_index=-100):
    was_training = model.training
    total_loss_sum = 0.0
    total_valid_targets = 0
    model.eval()
    try:
        for inputs, targets in held_out_batches:
            inputs, targets = inputs.to(device), targets.to(device)
            logits, no_loss = model(inputs)  # read-only; no training targets branch
            assert no_loss is None
            flat_targets = targets.reshape(-1)
            total_loss_sum += torch.nn.functional.cross_entropy(
                logits.reshape(-1, model.config.vocab_size),
                flat_targets,
                ignore_index=ignore_index,
                reduction="sum",
            ).item()
            total_valid_targets += int(
                flat_targets.ne(ignore_index).sum().item()
            )
    finally:
        model.train(was_training)
    if total_valid_targets == 0:
        raise ValueError("held-out evaluation has no valid targets")
    return total_loss_sum / total_valid_targets

\inf\mathcal L_{\mathrm{fixed\ batch}}=\frac{-\log(1/2)-\log(1/2)}{6}=\frac{\ln(2)}{3}\approx0.231

\mathcal L_{\mathrm{train}}\downarrow\ \not\Rightarrow\ \mathcal L_{\mathrm{held\text{-}out}}\downarrow

Knowledge check

Which stage can evaluate generalization?

17. Project B: Train on Independent Documents, Validate and Explain Results

The package's data/documents contains eight original English training texts and three independent validation texts. Both cover observation, learning and everyday activities, reducing the confound of entirely unrelated domains. The corpus is tiny and stylistically narrow, not a language-capability benchmark. Absence of exact validation text in training does not establish absence of all near-duplicates or biases.

Scroll horizontally to view all columns.

Course data table
Condition	This experiment's setting	Why state it explicitly?
Input unit	30 fixed characters: a–z, space, period, comma and newline	The alphabet is independently specified, not expanded from validation data.
Model	Same MiniGPT class; T_max=24, C=32, H=4, two blocks	This is a new configuration, not the 520-parameter five-token baseline.
Structure	Explicit causal attention, Pre-Norm, GELU FFN, untied weights, no dropout	Same mechanisms as the main path; explicitly construct a new model for the new data.
Training	CPU, seed=7, batch=4, AdamW lr=0.003, decay=0.01, gradient-norm cap 1	Record actual conditions; do not promise a runtime or convergence value.
Evaluation	Every 20 updates, evaluate all fixed train/validation windows with eval + no_grad	Avoid comparing changing random evaluation batches or altering training's sampling sequence.

Split documents before creating windows. T=24 requires 25 consecutive characters: the first 24 are inputs, the last 24 targets. stride=24 gives non-overlapping target positions between adjacent windows. A tail too short for another 25-character window is unused; report the actual target count. Windows cross neither documents nor train/validation boundaries.

text

Illustrative document beginning (the program splits characters, not words):
25 raw characters   [c0,c1,...,c24]
24 input characters [c0,c1,...,c23]
24 targets          [c1,c2,...,c24]
The next window starts at c24; its first target is c25.

Recall Week 6: supplying targets does not let the model read them. The output at t uses inputs only through t within the window. Scoring many positions provides parallel supervision; it does not remove the causal mask.

bash

cd course_examples
python week12_generalization.py --steps 200 --eval-every 20 --seed 7 --output runs/first

This is a runnable program. Install PyTorch following the English package README first. The output directory must not already exist; use a new path such as runs/second for another run. It will not overwrite an earlier record. No external corpus download or GPU is required.

Scroll horizontally to view all columns.

Course data table
Generated file	How to read it
config.json	Records Python/PyTorch, CPU, seed, model, vocabulary and evaluation settings.
data_report.json	Per-document character/window counts, text hashes, full-containment checks and 48-character overlap counts.
loss.csv / loss.svg	Every record comes from an actual forward. Both curves evaluate fixed windows in eval mode.
samples.json	Continuations for three fixed prompts. They may repeat or contain broken spelling; do not report only the best one.
inference.pt	Model configuration, vocabulary, weights and explicit format identity. Intended for inference loading, not exact training resumption.
experiment_record.md	Actual initial/final losses and loading differences, plus interpretation for you to complete.

How should validation loss be aggregated? If one batch has 48 valid targets with mean loss 2 and another has 24 with mean 1, the combined mean is (48×2+24×1)/(48+24)=1.6667, not 1.5. The program sums batch means multiplied by their valid-target counts, then divides by the total count—the same convention as Week 11.

If both curves fall, average prediction improves on both datasets in this limited experiment. If training improves while validation worsens, inspect duplication/splits, alignment, mode and sample size before concluding overfitting. One rebound is not decisive, and improved validation loss does not prove factuality, reasoning or open-domain generalization.

bash

python week12_generalization.py --generate-only runs/first/inference.pt --prompt "a " --new-tokens 80

Loading reconstructs the saved configuration and character table. Do not give this checkpoint to the five-token model's strict loader: the formats and vocabularies differ explicitly. The program compares logits before/after loading for the same input; it does not claim exact replay of interrupted random training.

Scroll horizontally to view all columns.

Course data table
Exercise stage	Change just one thing	Evidence
A: Add sentences within the vocabulary	Use only the five known tokens; keep three tokens per sentence	Print inputs/answers; verify IDs and shapes
B: Add genuinely independent material	Split train/validation by source first	Traceable provenance; no validation data used for updates
C: Add a token	New vocabulary version, V and model	Resize embedding/head together; explicitly reject the old checkpoint
D: Longer context	New block_size and length handling	Recheck masks, shifting and causality

A is still a mechanism exercise. Discuss unseen-text performance only with sufficient data and an independent split. Submit observations at each stage instead of changing tokenizer, model capacity and trainer all at once.

Knowledge check

Why does this experiment provide more evidence than three fixed sentences without proving genuine language understanding?

18. Characters, Bytes, Words and Subwords: A New Unit Changes the Interface

Subword/BPE reuses common pieces within finite V, balancing sequence length and vocabulary coverage. Tokenizer training learns merges; later encoding applies them as frozen rules.

Scroll horizontally to view all columns.

Segmentation here illustrates units only. Exact BPE pieces depend on learned merges.
unit	Illustration using “我喜欢AI，AI也喜欢猫。” (“I like AI, and AI also likes cats.”)	Advantage	Cost
character	One Unicode code point at a time	Easy to inspect	Long words or English passages may require many positions
word	Words / specified boundaries	Shorter sequences	Unknown words and segmentation rules
UTF-8 byte	Each byte value 0..255	Reconstructs encoded text when normalization and byte handling preserve it	Chinese code points typically use multiple UTF-8 bytes
BPE/subword	Depends on learned merges	Tradeoff between coverage and length	Merge rules need versioning

P_{\mathrm{input/output\ tables}}\approx2VC\quad\text{for untied token embedding and bias-free LM head}

Knowledge check

Which two layer boundaries must change when V changes?

19. Scale the Model: Change One Explainable Factor at a Time

First check the fixed model's pipeline and learning behavior. Then change one factor and record configuration, seed, data split, updates, loss and samples.

Scroll horizontally to view all columns.

Course data table
lever	Interface / constraint	Main effect
block_size T_max	Expand position table and causal mask together	Attention-score memory/work grows approximately as T²
n_embd C	All residual paths must share one width	Projection/FFN parameters and compute grow roughly as C²
n_head H	C mod H=0，d_head=C/H	attention routing partition
n_layer	ModuleList depth and state keys change	Sequential transformations; memory/compute grow roughly linearly with depth
corpus/tokenizer	Version the mapping, V and data split	Training-signal coverage and embedding/head sizes

d_{\mathrm{head}}=C/H\in\mathbb N,\qquad \#\mathrm{attention\ scores}=BHT^2

Changing block_size makes position_embedding and causal_mask shapes incompatible with the old checkpoint. Changing V affects both ends of the model; width/depth affect many state keys/shapes. Design migrations explicitly.

Knowledge check

What is the core integer constraint in this multi-head implementation?

20. Project Checklist: From Inputs and Targets to Evidence of Learning

Scroll horizontally to view all columns.

Course data table
group	Evidence to observe
Tokenizer	Five-token round trip; unknown inputs rejected; long IDs 0..4; exact SHA-256 identity
Batch	Exact raw/input/target arrays, [3,2] shapes and six next-token pairs
Model	[3,2,5]；520 params；two registered blocks；stable member names
Gradient/update	finite loss/expected grads；one step changes parameter；count increments after success
Checkpoint	strict schema/config/tokenizer/untied/AdamW checks；restored eval logits equal reference
Causality	Change only a future token; earlier-position logits remain close

python

model.eval()
a = torch.tensor([[0, 1]], dtype=torch.long, device=device)
b = torch.tensor([[0, 4]], dtype=torch.long, device=device)
with torch.no_grad():
    logits_a, _ = model(a)
    logits_b, _ = model(b)
torch.testing.assert_close(logits_a[:, 0, :], logits_b[:, 0, :])

p_\theta(x_{t+1}\mid x_{\le t},x_{>t})=p_\theta(x_{t+1}\mid x_{\le t})

Tokenizer/batch failure: stop
Forward/loss failure: stop before backward
Gradient/update failure: diagnose before enlarging the model
Checkpoint round-trip failure: reject incompatible state
Causality failure: inspect masks, slices and axes first
Generation shape/probability failure: inspect crop/last/sample/append

Knowledge check

Which experiment directly checks that position 0 did not read position 1?

21. How Weeks 1–12 Connect: Each Week Owns Part of the Explanation

Scroll horizontally to view all columns.

Course data table
Week	Responsibility in the final system	Object in this trace
1	Numbers, functions, vectors/matrices and shapes	[3,2], [3,2,4] and matrix multiplication
2	Loss and gradient descent	Scalar CE and update direction
3	neurons/nonlinearity	The FFN's Linear→GELU→Linear
4	chain rule/backprop	loss.backward() leading to parameter.grad
5	tensor axes/PyTorch	B, T, C, H, d_head, V and broadcasting
6	token IDs、embedding、next-token CE	V=5 corpus and six shifted tasks
7	causal Q/K/V	[B,H,T,T] scores/mask/value aggregation
8	residual、norm、FFN Block	Two Pre-Norm blocks preserving [3,2,4]
9	tokenizer/data protocol	Freeze units/IDs; keep w09-readable-v1 separate
10	canonical MiniGPT ownership	GPTConfig、stable members、520 params
11	training/validation/checkpoint/generation lifecycle	AdamW validators、strict restore、stable sampling
12	Assembly and observability	The same trace from text to generation after restoration

Weeks 1–5: mathematics and tensor computation
Weeks 6–9: language objective, context and data identity
Weeks 10–11: architecture and lifecycle state
Week 12: assembly, restoration and diagnosis in one evidence chain

\theta\xrightarrow{\mathrm{forward}}Z\xrightarrow{\mathrm{cross\ entropy}}\mathcal L\xrightarrow{\mathrm{chain\ rule}}\nabla_\theta\mathcal L\xrightarrow{\mathrm{AdamW}}\theta'

Knowledge check

Which week explains why attention scores have T×T axes?

22. Final Mental Model: A Stateful Conditional-Probability System

The tokenizer defines a discrete protocol. Embeddings and a causal Transformer turn permitted left context into representations. The output head scores candidates; Softmax turns the selected final row into a conditional distribution. Training changes θ; generation grows history without changing θ.

Scroll horizontally to view all columns.

Course data table
lane	Complete trace	Long-lived object changed
training	text→IDs→[3,2]→[3,2,5]→[6,5]+[6]→loss→grads→step	θ、AdamW state、completed_updates
generation	prompt→crop→logits→last [B,5]→sample [B,1]→append	Caller history; θ/optimizer unchanged
checkpoint restore	validated serialized state→model/optimizer objects	Explicit restoration of long-lived state, not learning

p_\theta(x_{1:L})=\prod_{t=1}^{L}p_\theta(x_t\mid x_{<t})

\theta'=\operatorname{Optimizer}(\theta,\nabla_\theta\mathcal L,\mathrm{optimizer\ state})

The full-sequence factorization below is general language-model notation. This five-token MiniGPT has no BOS and rejects empty input, so actual generation starts with a nonempty prompt and models subsequent tokens conditional on it. The first prompt token is not predicted from empty input. This implementation also conditions each new prediction on only the final two tokens retained in its cropped context.

Knowledge check

What stays fixed and what grows during one generation call?

23. Final Understanding Check: Trace a Symptom to Its Boundary

The error is at the generation boundary. Select logits[:,-1,:] [B,5], apply stable temperature/top-k/Softmax and use multinomial to obtain torch.long next_id [B,1]. Append that ID, not a five-dimensional probability vector.

\mathbb R^{B\times V}\xrightarrow{\mathrm{sampling}}\mathbb N^{B\times1}\xrightarrow{\mathrm{append}}\mathbb N^{B\times(L+1)}

torch.load(..., map_location="cpu") relocates serialized tensors without changing vocabulary meaning. Schema/version, ordered tokenizer artifact/hash/policy, exact GPTConfig, untied policy and model state keys/shapes must still match. Optimizer-state resumption also requires matching AdamW class/group/order/state and completed_updates.

Knowledge check

Why can generation not directly append [B,5]?

24. Finish the Course: Explain One Learning Update and One Generation Step

Learning searches for parameters that reduce average next-token prediction error under specified examples, a model and a loss. Embeddings, Q/K/V, residuals, FFNs and AdamW serve the same goals: computable conditional scores, differentiable error and adjustable parameters.

\boxed{\mathrm{Learning}=\text{finding parameters }\theta\text{ that minimize measured prediction loss}}

\theta^*=\arg\min_\theta\mathbb E_{(x,y)\sim\mathcal D}[\mathcal L(f_\theta(x),y)]

Define the text/token/data contract
Verify tensor shapes and causal forward behavior
Use loss and gradients to complete updates
Save and strictly restore compatible state
Assess usefulness and limitations through held-out evidence and target-free generation

Knowledge check

What are the three ingredients of MiniGPT's objective?