07-02 Stack Blocks Into MiniGPT

Why this matters

Stacking blocks creates the full forward path from embeddings to vocabulary logits.

Intuition first (no jargon)

Compose embedding, positional signal, repeated blocks, and final projection without breaking contracts.

Paper grounding

• The paper stacks N = 6 identical layers in both encoder and decoder.
• All sublayers and embedding layers produce vectors of dimension d_model, which enables residual connections.

Worked example

• Inputs: tokens [[4, 2, 9], [1, 7, 3]] with shape [B][T] = [2][3]
• Shapes:
  • token + position embeddings -> [2][3][dModel]
  • after each transformer block -> [2][3][dModel]
  • final LM head -> logits [2][3][V]
• One computed step:
  • if block input for batch 0, time 1 is vector h, the block returns vector h' of same length dModel
• Output: logits are produced for every token position, not only the last one.

Code walkthrough

js
export function miniGPTForward(tokens, params) {
  // token embedding + position
  // N transformer blocks
  // final norm + lm head
}

Your task

Implement miniGPTForward(tokens, params).

• Accept tokens shaped [B][T].
• Produce logits shaped [B][T][V].
• Reuse shared modules from previous lessons.
• Stage implementation:
  • Step 1: implement and test B = 1 path.
  • Step 2: generalize loops to B > 1.
  • Step 3: verify output shape for B = 2, T = 3.

Common mistakes

• Projected only final time step

Precision note

Target: architecture correctness and shape consistency at each stage.

Hints

• Start with B = 1 path, then generalize.
• Ensure each block gets a causal mask for T.
• Keep batch loop simple for readability.

Check your thinking

1. Why is output needed at every time step?
2. What changes when stacking more blocks?
3. Why keep module boundaries clear?

Stretch (optional)

Add parameter count utility for model-size reporting.

Likely test focus

• Correct output shape [B][T][V].
• Deterministic output for fixed params.
• Proper use of causal mask in each block.

What should improve

You can now produce logits for every batch/time position from a full stacked model.

Bridge to next lesson

Next lesson: batching and one training step.