06-03 Residual and LayerNorm

Why this matters

Residual paths preserve signal, and normalization stabilizes scale across stacked sublayers.

Intuition first (no jargon)

Add a shortcut and normalize around each sublayer to keep deep composition trainable.

Paper grounding

• Section 3.1 applies a residual connection around each sublayer and then layer normalization.
• The canonical form is LayerNorm(x + Sublayer(x)).

Code walkthrough

js
export function residualAdd(x, fx) {}
export function layerNorm(x, gamma, beta, eps = 1e-5) {}

Your task

Implement residual addition and layer normalization.

• Add vectors elementwise in residualAdd.
• Normalize each token vector to zero-mean, unit-variance.
• Apply scale (gamma) and shift (beta).

Hints

• Compute mean and variance per token vector.
• Use small eps to avoid divide-by-zero.
• Validate matching vector lengths.

Check your thinking

1. Why can deep models fail without residuals?
2. Why normalize per token rather than across batch here?
3. What does gamma and beta allow the model to do?

Stretch (optional)

Implement both pre-norm and post-norm block variants.

Likely test focus

• Correct normalization on controlled vectors.
• Correct residual output values.
• Stable behavior for near-constant vectors.

What should improve

You now have the residual-plus-normalization structure used in Transformer blocks.

Bridge to next lesson

Next lesson: compose the full block forward pass.