Transformer Block Interview Cheat Sheet

§0 The Basic Parts of a Block

A Transformer block organizes attention and FFN with residual connections. In the serial structure the FFN reads the residual stream after the attention update; in the parallel structure the two have no ordering dependency. The decoder layer of the original encoder-decoder also contains a cross-attention. Normalization, KV sharing, and model structure each have their own choices.

• A block's definition contains only its own sublayers: final norm, local/global interleaving, and dense/MoE layer placement are decided by the stack-assembly code and belong to no block.
• GPT-2 is already Pre-LN: from GPT-2-style to Llama-style, what actually changed is the normalization type, the positional encoding's owning level, the FFN structure, and the KV-head configuration.
• MHA, MQA, GQA, and MLA are coexisting design points: the historical order is MHA→MQA→GQA, the compression-strength order is MHA→GQA→MQA, and MLA is an independent low-rank latent branch.
• The KV cache mainly tracks $n_\text{kv}$, not $n_q$: MQA/GQA do not reduce the Q projection's compute.
• Dense FFN and MoE are two choices on the capacity axis: not a "next generation inevitably replaces the previous" ordering.
• The only structural difference between cross-attn and self-attn is where Q/K/V come from: everything else — scaled dot-product, multi-head split, softmax — is identical.

§1 A Modern Block: Forward, Parameters, and Code

1.1　Assemble a Llama-3-like block, then swap in MoE

A concrete, complete, runnable modern decoder-only block looks like this:

1. Normalization: Pre-RMSNorm, twice — once before attention, once before the FFN.
2. Attention: causal self-attn, RoPE applied internally to Q/K, $n_q$ query heads grouped to share $n_\text{kv}$ KV heads — that is GQA, degenerating to MHA when $n_\text{kv}=n_q$; K is cached after rotation, V is never rotated, no bias.
3. FFN: SwiGLU gated FFN, $h \approx 8d/3$ rounded to a hardware-friendly multiple, no bias.
4. Single-block assembly: $x=x+\text{Attn}_\text{RoPE,GQA}(\text{RMSNorm}(x))$, $x=x+\text{SwiGLU}(\text{RMSNorm}(x))$ — this is the complete definition of one block; repeat it $N$ times to get the whole decoder stack.
5. Stack-level closure: after all $N$ blocks, apply one final RMSNorm before the LM head: $x_\text{out} = \text{RMSNorm}\big(\text{Block}_N(\cdots\text{Block}_1(x_0))\big)$. The final norm belongs to stack assembly, not to any block in steps 1–4.

Strictly this is a "Llama-3-like core block": exactly reproducing a given checkpoint also requires matching head count, KV-head count, intermediate size, RoPE base, bias settings, and dtype behavior. Compute the parameter count once with the teaching sizes $d=96$, $n_q=8$, $n_\text{kv}=2$, gated hidden $=256$: the GQA attention part has $2d^2(1+n_\text{kv}/n_q)=23{,}040$ matrix parameters, the SwiGLU FFN part has $3dh=73{,}728$ matrix parameters, and each of the two RMSNorms has only $d=96$ per-channel scale parameters. A single block totals about $23{,}040+73{,}728=96{,}768$ matrix parameters, with the FFN taking the vast majority. This is also why the FFN hidden size affects total parameters more than the KV-head count.

Swap the FFN for MoE and touch nothing else:

$$x=x+\text{Attn}_\text{RoPE,GQA}(\text{RMSNorm}(x)), \qquad x=x+\text{MoEFFN}(\text{RMSNorm}(x))$$

MoEFFN's external main-activation tensor interface is still [B,T,d] -> [B,T,d], and not a single line of ModernDecoderBlock changes; §1.4 [D09] verifies this directly. What changes are two other things: training usually returns extra router/load-balancing quantities, and distributed deployment usually introduces expert parallelism.

1.2　End-to-end forward: the complete data flow of a modern decoder-only block

Take the teaching sizes $d=96$, $n_q=8$, $n_\text{kv}=2$, gated hidden $=256$. The complete forward data flow of a modern decoder-only block is:

$$x \in [B,T,d] \;\xrightarrow{\text{RMSNorm}}\; h_1 \;\xrightarrow{\text{Q/K/V projection + reshape}}\; Q\in[B,n_q,T,d_h],\;K,V\in[B,n_\text{kv},T,d_h] \;\xrightarrow{\text{RoPE}}\; Q',K' \;\xrightarrow{\text{GQA attention}}\; O\in[B,T,d] \;\xrightarrow{\text{residual add}}\; u = x + O$$

$$u \;\xrightarrow{\text{RMSNorm}}\; h_2 \;\xrightarrow{\text{gate/up projection}}\; \text{SiLU}(\text{gate}(h_2))\odot\text{up}(h_2) \;\xrightarrow{\text{down projection}}\; F\in[B,T,d] \;\xrightarrow{\text{residual add}}\; y = u + F$$

Repeat this block $N$ times and then apply one stack-level final RMSNorm to get the full decoder-only backbone: $y_\text{final} = \text{RMSNorm}\big(\text{Block}_N(\cdots(\text{Block}_1(x_0)))\big)$.

This data flow takes different concrete forms under three execution modes:

The three modes share the same RMSNorm→RoPE→attention→residual code path, differing only in the actual Q/K/V lengths, the mask shape, and whether the cache is read/written. That is why §1.4 [D04] can use one implementation to verify that full-forward and cached decoding agree numerically; if the two paths used different code, that test would lose its cross-validation meaning.

1.3　Parameter-count and KV-cache formula accounting

FFN parameters, ignoring bias: dense (ReLU/GELU) is $8d^2$ over two matrices; gated (SwiGLU/GeGLU) is $3dh$ over three matrices. Setting them equal gives $h=8d/3$ — a common default computed from parameter matching, not an architectural law. Real Llama-family implementations usually start from $4d$, multiply by $2/3$, then round to hardware-friendly multiples (with an optional multiplier), so the intermediate size can deviate noticeably from $8d/3$.

Attention parameters, ignoring bias: MHA's four Q/K/V/O projections total $4d^2$; GQA gives $2d^2(1+n_\text{kv}/n_q)$, degenerating back to $4d^2$ at $n_\text{kv}=n_q$ and reaching MQA's minimum at $n_\text{kv}=1$. Both formulas assume $d=n_qd_h$, identical head dim for Q/K/V, a $d\times d$ output projection, and no bias — adjust them accordingly when these assumptions fail.

Packed projection: fusing the three Q/K/V Linears, or SwiGLU's gate/up Linears, into one big Linear is a kernel-level equivalent rearrangement — with row-wise concatenated weights the results should agree at the corresponding precision, changing no architectural semantics.

KV cache: the table below gives element counts per token per layer, not bytes. Full memory usage further multiplies by batch size $B$, sequence length $L$, layer count $N_\text{layer}$, and bytes per element — usually 2 bytes for fp16/bf16, 1 byte for fp8.

1.4　From-scratch implementation and executable verification: code/transformer_block.py

The full runnable script is code/transformer_block.py (pure PyTorch; all 9 demos [D01]–[D09] finish in seconds on CPU). Four entry classes correspond to this tutorial's four "block eras": VanillaEncoderLayer2017, VanillaSeq2SeqDecoderLayer2017 (§2), GPT2StyleDecoderLayer (§3), ModernDecoderBlock (§4/§1.1, with ffn= swappable among DenseFFN/GatedFFN/MoEFFN). The body shows the three most central snippets; the remaining demos are standalone functions in the script.

[D02] Residual-topology test (zero out the sublayer output: Pre-LN must return exactly $x$, Post-LN must return $N(x) \ne x$):

def zero_sublayer(_x):
    return torch.zeros_like(_x)

post_ln_out = ln(x + zero_sublayer(x))       # y = N(x + F(x)) -> y = N(x)
pre_ln_out  = x + zero_sublayer(ln(x))       # y = x + F(N(x)) -> y = x

assert torch.allclose(pre_ln_out, x, atol=1e-6)          # Pre-LN: exact identity
assert torch.allclose(post_ln_out, ln(x), atol=1e-6)     # Post-LN: exactly N(x)
assert not torch.allclose(post_ln_out, x, atol=1e-4)     # and generally != x

[D04] Full forward vs cached decoding (under eval() with no dropout, the full-sequence forward and token-by-token incremental decoding must agree numerically — jointly checking the causal mask, RoPE position offset, and cache append):

causal = build_causal_mask(torch.arange(T), torch.arange(T))
out_full = block(x, cos, sin, mask=causal)                  # compute the whole sequence at once

cache = KVCache()
outs = []
for t in range(T):
    q_pos, k_pos = torch.tensor([t]), torch.arange(t + 1)
    mask_t = build_causal_mask(q_pos, k_pos)                 # new token sees all of 0..t
    outs.append(block(x[:, t:t+1, :], cos, sin, mask=mask_t,
                       kv_cache=cache, start_pos=t))
out_cached = torch.cat(outs, dim=1)
assert torch.allclose(out_full, out_cached, atol=1e-4)
assert cache.k.shape[1] == n_kv_heads                        # cache head count never grows to n_q_heads

[D08] Parallel vs sequential dependency (use a forward hook to capture the FFN's actual input tensor and exactly match it against an independently recomputed expected tensor — not the fragile "two random numbers are unequal"):

captured = {}
seq_blk.ffn.register_forward_hook(lambda m, i, o: captured.setdefault("x", i[0]))
seq_blk(x, causal)
a, _ = seq_blk.attn(seq_blk.norm1(x), mask=causal)
assert torch.allclose(captured["x"], seq_blk.norm2(x + a), atol=1e-5)   # sequential: FFN sees the "updated" stream

captured.clear()
par_blk.ffn.register_forward_hook(lambda m, i, o: captured.setdefault("x", i[0]))
par_blk(x, causal)
assert torch.allclose(captured["x"], par_blk.norm(x), atol=1e-6)       # parallel: FFN sees the "original" stream

Key invariants the script must satisfy (one per [D01]–[D09], described here by "verification intent"; the exact assertions are as implemented in the script):

1. [D01] Shape contract: all four entry classes preserve the [B,T,d] residual width; cross-attn still correctly returns length-$T$ output when $T\ne S$, and must verify that memory is actually read (changing memory changes the output), not just the shape.
2. [D02] Residual-topology test: with sublayer outputs zeroed, Pre-LN-style blocks return exactly $x$; Post-LN-style blocks return $N(x)$, generally not equal to $x$; the two sublayers' LNs must be compared separately — one $N(x)$ must not be used to certify itself.
3. [D03] No causal leakage: perturbing only tokens after position $t{+}1$ must leave outputs at positions $0..t$ untouched; confirm this invariant is actually contributed by the attention branch.
4. [D04] Full forward vs cached decoding: under eval() with no dropout, the full-sequence forward and token-by-token incremental decoding agree numerically at all positions, and the cache head dimension stays $n_\text{kv}$ throughout — the most direct way to hunt causal-mask / RoPE-offset / cache-ordering bugs, though not the only test that must cover a nonzero attention-branch contribution.
5. [D05] MHA/MQA/GQA degeneration: at $n_\text{kv}=n_q$ the general implementation matches an independently hand-written plain-MHA reference exactly; $n_\text{kv}=1$ (MQA) and intermediate values (e.g. $n_\text{kv}=2$) must each be compared against an independent repeat_interleave reference — testing only the $n_\text{kv}=n_q$ degenerate case is not enough.
6. [D06] Parameter formulas: at $d=96$, dense FFN (hidden $=4d$) and gated FFN (hidden $=8d/3$) both have exactly 73,728 matrix parameters; GQA/MHA attention matrix parameters equal the $2d^2(1+n_\text{kv}/n_q)$ and $4d^2$ formulas exactly; illegal configurations where $d_\text{model}$ is not divisible by the head count must fail at construction time.
7. [D07] Packed-projection numerical equivalence: packed QKV (and packed gate+up) with identical weights match three (or two) separate Linears numerically (numerically equivalent within tolerance, not necessarily bitwise identical).
8. [D08] Parallel vs sequential: forward hooks capture both attention's and the FFN's actual inputs; the sequential block matches $N_2(x+\text{Attn}(N_1(x)))$ exactly, and both branches of the parallel block match $N(x)$ exactly.
9. [D09] Backward smoke test: all dense-block parameter gradients are finite; the MoE-swapped block must iterate over all experts not unused in this batch and check nonzero gradients, while router/attention/norm and all other parameters also receive gradients normally.

The actual output of running python3 code/transformer_block.py (D10-D15 are targeted regression tests, covering constructor argument validation, KV-cache consistency, cross-attention combination restrictions, mask safety and broadcasting, default causal behavior, and RoPE-cache device/dtype handling):

[D01] shapes (enc/dec2017/gpt2/modern) = [torch.Size([2, 7, 96]), torch.Size([2, 7, 96]), torch.Size([2, 7, 96]), torch.Size([2, 7, 96])] (expect 4x (2,7,96)); cross-attn ran with Tq=7 != S=5; changing memory changes cross-attn output (memory is actually read, not ignored) = True  PASS
[D02] formula-level: Pre-LN(F=0)==x: True; Post-LN(F=0)==N(x): True (and != x: True); class-level: zeroed ModernDecoderBlock==x: True, zeroed Post-LN encoder layer != x: True, == ln2(ln1(x)) exactly: True  PASS
[D03] perturbing tokens after t=3: positions 0..3 unchanged = True, positions 4..5 changed = True; perturbing position 0 changes positions 1..5 (attention actually mixes forward, not a dead branch) = True  PASS
[D04] full-forward vs token-by-token cached decode: max|Δ| = 4.77e-07 (tol 1e-04, numerically equivalent, NOT bit-identical); cache CONTENT vs independently projected+rotated K/V: max|ΔK|=3.58e-07, max|ΔV|=4.77e-07; final cache shape head_dim=2 (expect 2) len=6 (expect 6)  PASS
[D05] GQA(n_kv=n_q) == independent reference MHA: max|Δ|=0.00e+00; GQA(n_kv=2, group=4) == independent repeat_interleave reference: max|Δ|=8.94e-08; MQA == same reference (n_kv=1): max|Δ|=8.94e-08, cache head_dim=1 (expect 1), output shape=(2, 5, 96), per-query-head outputs distinct (not collapsed) = True  PASS
[D06] FFN params: dense(4d)=73728, gated(8d/3)=73728 (both expect 73728); attn params: GQA=23040 (formula 2d^2(1+n_kv/n_q)=23040), MHA=36864 (formula 4d^2=36864)  PASS
[D07] packed QKV vs separate: max|Δ|=4.77e-07; packed gate+up vs separate: max|Δ|=0.00e+00  PASS
[D08] sequential: attn input == norm1(x): True, FFN input == norm2(x+attn_out) (updated stream): True (and != norm2(x) alone: True); parallel: attn input == norm(x): True, FFN input == norm(x): True, attn and FFN inputs are the literal SAME tensor: True  PASS
[D09] dense block: all grads finite = True, all non-zero = True; MoE tokens_per_expert=[1, 0, 2, 0] (<4 tokens, so >=1 expert unused by pigeonhole); touched experts non-zero+finite grad, untouched experts no/zero grad: True; router gate non-zero+finite grad: True; rest-of-block (attn+norms) non-zero+finite grad: True  PASS
[D10] d_model % n_q_heads != 0 raises ValueError: True; n_q_heads % n_kv_heads != 0 raises ValueError: True; RoPE + odd head_dim raises ValueError: True; MoEFFN(top_k=2) raises NotImplementedError (no silent top-1 downgrade): True  PASS
[D11] inconsistent start_pos vs non-empty cache raises ValueError: True; consistent start_pos does not raise: True; cache.reset() clears state: True; append() with mismatched batch/shape raises ValueError: True  PASS
[D12] cross-attention + RoPE raises ValueError: True; cross-attention + KV cache raises ValueError: True; plain cross-attention (no RoPE/cache) still works, shape=(1, 4, 32): True  PASS
[D13] fully-masked query row produces finite output (no NaN): True; key_padding_mask correctly isolated per-batch-sample when B==n_kv_heads (2): max|Δ| vs per-sample reference=1.79e-07 -> True; same check when B==Tq (3): max|Δ|=1.64e-07 -> True  PASS
[D14] GPT2StyleDecoderLayer: omitted causal_mask == explicit causal_mask: True, default is genuinely causal (no future->past leakage): True; ModernDecoderBlock: omitted causal_mask == explicit causal_mask: True  PASS
[D15] build_rope_cache(dtype=float64) produces float64 cos/sin: True; apply_rope auto-casts a mismatched (float32) cache to a float64 input's dtype (no crash, correct output dtype): True; matches a natively-float64-built cache: True (max|Δ|=0.00e+00)  PASS

all transformer block sanity checks passed ✓

§2 The Original Encoder-Decoder

The original Transformer (Vaswani et al., Attention Is All You Need, arXiv 1706.03762, 2017, NeurIPS 2017) is an encoder-decoder architecture: the encoder encodes the source sequence into a set of representations, and the decoder autoregressively generates the target sequence conditioned on them. Positional encoding uses fixed sinusoidal PE, added directly to the token embeddings before entering the encoder/decoder stacks; it belongs to no block and is input processing outside the blocks. The paper's original configuration is $d_\text{model}=512$, $d_\text{ff}=2048$ (4× expansion), 8 heads, 6 encoder + 6 decoder layers; this tutorial demonstrates with smaller sizes such as $d=96$, which is a teaching configuration, not the paper's — do not conflate the two.

The encoder layer and the decoder layer differ in both the number and the kinds of sublayers:

The decoder's three sublayers run in the fixed order masked self-attn → cross-attn → FFN, each wrapped in its own Post-LN. Cross-attn's Q, K, V sources are asymmetric: Q comes from the decoder's own in-progress stream, length $T$; K and V come from the encoder output, length $S$, and $T$ and $S$ can differ.

Walking one concrete shape through, with teaching sizes batch $B=2$, source length $S=5$, target length $T=7$, $d=96$: the encoder input is [B, S, d] and remains [B, S, d] after several encoder layers, passed as memory to every decoder layer; the decoder input is [B, T, d], and in the masked self-attn stage Q/K/V are all computed internally from [B, T, d], with a causal mask of size $T\times T$; in the cross-attn stage Q is still [B, T, d] but K, V come from the encoder memory with shape [B, S, d], so the attention score matrix is $T\times S$ rather than square, with softmax along the $S$ dimension; after the output projection it returns to [B, T, d], matching the self-attn sublayer's output shape, so it can proceed into the FFN sublayer. $T \ne S$ does not affect this data path at all — this is what "cross-attn decouples Q length from K/V length" concretely looks like.

VanillaEncoderLayer2017 (2 sublayers) and VanillaSeq2SeqDecoderLayer2017 (3 sublayers) are two independent classes in §1's code. Writing them as one "generic block class + parameter switches" is workable engineering but hides the structural fact that the decoder has a whole extra cross-attn sublayer compared with the encoder.

§3 The Three Architecture Families and GPT-2→Llama

3.0　encoder-only / decoder-only / encoder-decoder

The core difference among the three is not "how many parameters" but that attention visibility, sublayer composition, and training objective change together as a bundle.

3.1　How decoder-only differs from the original decoder layer

At the single-layer structural level, a decoder-only block really is the original decoder layer with cross-attn removed: masked self-attn → cross-attn → FFN, three sublayers, becomes causal self-attn → FFN, two sublayers.

At the whole-model level an entire encoder/memory path is missing as well: source/prompt and target go from "two-stream conditional generation" to "single-stream causal modeling", with prompt and generated content concatenated into one causal sequence. What changes is the information-flow topology and the sequence factorization, not that the task was replaced by a different problem.

Some multimodal/retrieval-augmented models reintroduce cross-attention on a decoder-only backbone; every "decoder-only has no cross-attn" statement in this tutorial is restricted to the pure decoder-only block, i.e. the two sublayers causal self-attn + FFN.

3.2　GPT-2-style to Llama-style

GPT-2 (Radford et al., Language Models are Unsupervised Multitask Learners, OpenAI technical report, 2019, no arXiv id) is already Pre-LN, and puts a final LayerNorm, denoted ln_f, at the end of the stack. The complete GPT-2 block recipe:

$$x = x + \text{Attn}\big(\text{LN}(x)\big), \qquad x = x + \text{FFN}\big(\text{LN}(x)\big), \qquad \text{(GELU dense FFN, MHA, bias=True, no cross-attn)}$$

The learned absolute positional embedding is added to the embedding and sits outside the stack, the same as the original sinusoidal PE; neither lives inside a block. RoPE, adopted by the LLaMA family, is the opposite: it lives inside attention, acting only on the Q/K dot product. RoPE was first proposed by Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv 2104.09864, 2021; LLaMA is a prominent adopter, not the proposer. Positional information is carried by an external embedding in GPT-2-style and by rotation inside attention in Llama-style; the two belong to different levels, so this is not a "seamless swap" between members of the same block. What actually changed is this:

§4 RMSNorm, RoPE, SwiGLU, and Model Composition

Llama-style — the recipe of Llama 2 and Llama 3 — fills each component with the concrete implementations below; attention is written as MHA first:

$$x = x + \text{Attn}_{\text{RoPE}}\big(\text{RMSNorm}(x)\big), \qquad x = x + \text{SwiGLU}\big(\text{RMSNorm}(x)\big)$$

• RMSNorm: re-scale only, no mean-centering, no bias: $y = \dfrac{x}{\sqrt{\text{mean}(x^2)+\epsilon}}\odot w$. It often reaches quality comparable to LayerNorm.
• RoPE: acts on Q and K inside attention; V is never rotated. Standard implementations usually cache the already-rotated K, and incremental decoding rotates new tokens at their true positions. The unrotated-K-cache variant is in §9 Q12.
• SwiGLU: $\text{down}\big(\text{SiLU}(\text{gate}(x)) \odot \text{up}(x)\big)$, a three-matrix gated FFN; GeGLU is the same structure with a GELU gate (Shazeer, GLU Variants Improve Transformer, arXiv 2002.05202, 2020). At similar budgets gated FFNs often deliver better quality — an empirical regularity, not a law.
• bias, dropout: Q/K/V/O and the three FFN matrices usually drop all bias; large-scale pretraining commonly removes dropout or reduces it drastically.
• final norm: one RMSNorm at the end of the stack, closing off the inflated residual stream. Under the independent-increment approximation the residual stream's variance grows roughly as $O(l)$; it is the standard deviation/RMS that goes as $\sqrt l$. Feeding the output head without this closure usually degrades training quality noticeably. It is a common configuration of the standard recipe, not mathematically indispensable.
• Numerical-precision boundary: computations sensitive to numeric range — RMS/variance reductions, softmax, final logits — are commonly done at higher precision such as fp32 and cast back to the bf16/fp16 activation dtype. This is part of assembly correctness, not quantization math.

Where a design decision lives determines who assembles it:

Weight tying, embedding scaling, the LM head's soft-cap, and the global dropout switch are part of the "complete LM recipe" but live outside the block — the token embedding and LM head belong to no Transformer block:

None of these four appears in the block classes of these examples: ModernDecoderBlock's forward receives only the residual stream x and never knows whether embedding scaling happened outside or whether the output head is tied.

§5 KV Sharing and MLA

Every step of autoregressive generation appends the new token's K, V to the cache and attends over the whole cache; the cache size directly determines how many requests one GPU can serve concurrently and how long a context it can hold. GQA/MQA reduce the KV-head count and simultaneously change the K/V projections' own parameter count and compute: the KV cache mainly affects inference memory/bandwidth, while K/V projection parameters are a fixed cost that also exists in training.

5.1　MHA / MQA / GQA: different compression ratios on the same axis

MHA is the baseline. **MQA (Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, arXiv 1911.02150, 2019) predates GQA (Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023), and GQA is the quality-memory compromise between MHA and MQA. MLA is DeepSeek's separate low-rank latent-compression branch**, not "reduce $n_\text{kv}$ once more". The four are coexisting design points on the "KV memory" design axis, not a historical ladder every model climbed in order.

• KV-cache size scales only with $n_\text{kv}$, independent of $n_q$ — MQA/GQA reduce the K/V projections' own parameters and compute, and also reduce KV-cache memory/bandwidth; but the Q projection's compute does not change with $n_\text{kv}$, so do not flatten this into "only saves memory/bandwidth".
• Standard implementations usually cache K after RoPE is applied; V is never rotated; in incremental decoding, the new token's Q and K get RoPE at the true position offset, and the already-rotated K in the historical cache is never recomputed.
• GQA/MQA caches store only $n_\text{kv}$ heads: broadcasting K/V to the $n_q$ Q heads may happen only inside the attention computation, via view/reshape/grouped einsum, and the broadcast tensor must never be written back to the cache. §1.4 [D05] turns this into an executable assertion.

With vanilla MHA, LLaMA-2-70B would need about 10 GiB of KV cache per sample at a 4096 context; switching to GQA ($n_\text{kv}=8$) drops it to about 1.25 GiB — one of the important reasons many large models adopt GQA over MHA.

5.2　MLA: the low-rank latent branch

MLA (DeepSeek-V2, May 2024, arXiv 2405.04434) compresses along the hidden dimension via low-rank projection: each token's K/V is squeezed into a latent vector with $d_c \ll n_q d_h$. It must decouple RoPE into a separately shared small-dimension component to preserve the "absorb the up-projection matrix into the query side" inference speedup. MLA's cache format is "store the latent $c_t^{KV}$ + one shared RoPE key", not "store fewer K/V heads".

GQA compresses the head count, MQA is GQA's extreme case ($n_\text{kv}=1$), and MLA compresses each token's representation dimension. The three are not three points on one scale but two different compression ideas.

§6 MoE FFN

MoE replaces the FFN with $N$ experts plus one router, and each token passes through only $k \ll N$ experts: total parameters go up, per-token activated parameters stay flat.

• The block's external main-activation tensor interface is unchanged: dense or MoE, it is [B, T, d] -> [B, T, d] and the residual add stays in place — the direct reason why swapping ffn for MoEFFN changes not a line of ModernDecoderBlock. But training usually returns extra router/load-balancing auxiliary losses or statistics, and distributed deployment usually introduces expert parallelism and all-to-all communication: an unchanged tensor shape does not mean unchanged training return values or system deployment.
• What changes is inside the FFN: a router, expert selection, possibly a shared expert, and training-time auxiliary balancing losses/bias updates are added. Total parameters and per-token activated parameters must be reported separately.
• Dense and MoE are two choices on the capacity axis: representative models such as Llama 1–3, the original Mistral 7B, and Gemma 1–3 use dense FFN; Mixtral, Llama 4, and DeepSeek-V2/V3 use MoE, trading the engineering complexity of communication/routing/load-balancing for larger sparse parameter capacity.

Mixtral (Jiang et al., Mixtral of Experts, arXiv 2401.04088, 2024) activates about 12.9B parameters per token and matches or beats Llama 2 70B on most benchmarks reported in the paper. Total parameters are far above 13B, and per-token compute is close to a dense model in the low tens of billions, but not strictly FLOPs-equivalent. DeepSeek-V3's specific load-balancing mechanism comes from Wang, Gao, Zhao, Sun & Dai (arXiv 2408.15664, 2024).

The full routing formulas, capacity factor, load-balancing loss, and token dropping are in moe_tutorial.md §2–§4.

§7 Residual Structure, Attention, and Execution

Residual connections, normalization, and branch scaling determine a block's computation structure. Numerical control inside attention adjusts scale, and local or global attention determines the visible range. FlashAttention, quantization, and LoRA serve execution optimization or fine-tuning. Most of these mechanisms can be combined; the concrete configuration is influenced by training stability, kernel support, and compatibility with existing checkpoints.

7.1　Residual topology, norm placement, and branch scaling

$A$ and $M$ are the attention and FFN sublayers, and $N_1,N_2,N_A,N_M,N_1^{\text{pre}},N_1^{\text{post}},N_2^{\text{pre}},N_2^{\text{post}}$ are all normalization operators. The four topologies are all written at "whole block" granularity:

$$\text{Post-LN}:\quad u = N_1\big(x + A(x)\big),\quad y = N_2\big(u + M(u)\big)$$

Vaswani et al. 2017's original Transformer uses this one: the normalization wraps the residual sum, repeatedly squeezing the trunk.

$$\text{Pre-LN serial}:\quad u = x + A\big(N_1(x)\big),\quad y = u + M\big(N_2(u)\big)$$

Explicitly adopted at least by GPT-2 and later the common form in modern LLMs: the trunk keeps a pure identity-add path, and the FFN reads the updated stream $u$.

$$\text{Pre-LN parallel}:\quad y = x + A\big(N_A(x)\big) + M\big(N_M(x)\big)$$

GPT-J and PaLM use this one: the ordering dependency is removed, and attention and FFN read the same $x$.

$$\text{branch pre+post}:\quad u = x + N_1^{\text{post}}\big(A(N_1^{\text{pre}}(x))\big),\quad y = u + N_2^{\text{post}}\big(M(N_2^{\text{pre}}(u))\big)$$

Gemma-2/3 use this one: the second normalization wraps only the branch output, and the trunk stays pure addition — unlike Post-LN.

Norm placement and serial/parallel dependency are two dimensions, not one axis; §1.4 [D02]/[D08] gives executable positive and negative proofs of all four topologies. branch pre+post means one normalization before and after each sublayer branch, in single-sublayer form $y = x + N_\text{post}\big(F(N_\text{pre}(x))\big)$. Gemma-2 uses this topology together with attention-logit/final-logit soft-capping; Gemma-3 keeps this norm topology but replaces soft-capping with QK-Norm.

Parallel is defined as "no intra-layer ordering dependency"; it does not require attn and FFN to share one set of norm parameters: two independent normalization modules also satisfy the definition as long as both read the original $x$. GPT-J (Wang & Komatsuzaki, EleutherAI open-source release, 2021, no formal paper/arXiv id) implements it with a single shared LayerNorm; PaLM (Chowdhery et al. (Google), PaLM: Scaling Language Modeling with Pathways, arXiv 2204.02311, 2022) continues the same idea.

Residual/branch scaling is another independent, optional assembly position: multiply the residual branch by a fixed or learnable scale coefficient. LayerScale multiplies each branch output by a per-channel, learnable, small-initialized coefficient; DeepNorm multiplies the residual branch by a depth-dependent fixed amplification factor, paired with a specific initialization so deep networks train more stably. Whether to scale, whether the scale is a fixed constant or a learnable parameter, and which branch it acts on are chosen separately from norm placement, and can also be used together with it.

7.2　attention scale, QK-Norm, logit soft-cap

Three mechanisms act at different positions on the same computation chain:

• attention scale: the standard $1/\sqrt{d_h}$ or a learnable temperature, controlling the overall magnitude of the softmax input — the most basic numerical control, present in almost every implementation.
• QK-Norm (Henry et al., Query-Key Normalization for Transformers, arXiv 2010.04245, 2020, EMNLP 2020 Findings): before the dot product, L2-normalize each head's Q and K separately, and replace the standard fixed $1/\sqrt{d_h}$ with a learnable scale/temperature; later models' implementation details may differ.
• logit soft-capping: after the dot product, apply a bounded transform like $c\tanh(z/c)$ to attention scores or final vocabulary logits, reining in the magnitude of scores/logits already computed.

The three act at different positions and control different quantities, and cannot be stacked blindly — enabling them together requires re-checking the overall scale, not simple addition.

7.3　local / sliding / global attention

This is the design axis of attention's connectivity structure / visible range (connectivity pattern) — not an "attention backend", and separate from positional-encoding schemes. Mistral (Jiang et al., Mistral 7B, arXiv 2310.06825, 2023) uses fixed-window sliding-window attention; Gemma-2 alternates local-window and global attention 1:1 across layers; Gemma-3 is closer to a periodic pattern of "several local layers then one global layer", about 5:1, not strictly one local layer then one global layer.

None of these change the block's outer residual interface; they only change "how far attention can see" inside. And the placement of "which layers are local, which are global" is itself a stack-assembly decision, not a single block's private attribute. Full sliding-window/StreamingLLM mechanics are in long_context_rope_yarn_mla_tutorial.md §10.

7.4　FlashAttention, quantization, and LoRA

These three mechanisms stack on top of an already-chosen architecture to accelerate it or adapt it to a deployment environment; they are not architectural components on the same level as GQA/MoE requiring a mutually exclusive choice:

• FlashAttention: changes only exact attention's computation order, IO scheduling, and memory-access pattern; it does not change attention's mathematical result and defines no new connectivity graph, so it stacks on MHA/GQA/MQA/MLA and any local/global pattern.
• Quantization: represents weights/activations in lower-precision numeric formats, stackable with any architectural choice above, at the cost of quantization error.
• LoRA: adds a pair of low-rank matrices beside trained weights for parameter-efficient fine-tuning, without changing the block's forward architecture.

§8 Model Comparison and Common Wrong Answers

8.1　Block-recipe comparison table for mainstream models

Conventions: "QK-Norm" refers strictly to explicit per-head Q/K normalization before the dot product (MLA's internal latent-projection normalization does not count); "branch pre+post" means $y=x+N_\text{post}(F(N_\text{pre}(x)))$, not the same as original Post-LN; the Attention column lists only the KV-head sharing scheme, with local/sliding/global patterns in the Notes column; final norm and local/global placement belong to stack assembly, and this table presents each model's overall recipe; none of these representative models' canonical recipes uses ALiBi as the primary positional scheme.

Three regularities readable from this table: (1) "Norm topology" and "norm type" almost always come bound together — Pre-LN with RMSNorm, Parallel Pre-LN with LayerNorm, branch pre+post also with RMSNorm; (2) all four of MHA/MQA/GQA/MLA appear among this same batch of representative models, supporting §5's "coexisting design points, not a historical ladder"; (3) "Parallel attn+FFN" is "yes" only in the GPT-J and PaLM rows — niche but genuinely in use, not an abandoned dead end.

The modern recipe has not fully converged: Pre-RMSNorm + RoPE + gated FFN + inference-friendly GQA is only the more common combination; Post-LN variants, LayerNorm, MHA, MQA, parallel residual, dense FFN, and MoE are all still used in production.

8.2　The twelve claims most likely to backfire

High-frequency wrong answers and the more accurate claims:

§9 30 High-Frequency Interview Questions

Three difficulty tiers; expand each for the answer key (the 10 questions that provide a discrimination action end with a pitfall note). L2/L3 are top-lab deep water (topology forks, assembly-axis independence, system correctness, cross-model recipe comparison).

L1 Must-Know

L2 Intermediate

L3 Advanced

📚 References

• Transformer (original encoder-decoder) — Vaswani et al., Attention Is All You Need, arXiv 1706.03762 (2017), NeurIPS 2017.
• GPT-2 (decoder-only, Pre-LN, learned absolute PE) — Radford et al., Language Models are Unsupervised Multitask Learners, OpenAI technical report (2019) (no arXiv id).
• GPT-3 — Brown et al., Language Models are Few-Shot Learners, arXiv 2005.14165 (2020), NeurIPS 2020.
• GPT-J (parallel residual) — Wang & Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, EleutherAI open-source release (2021) (no formal paper/arXiv id).
• PaLM (parallel residual, MQA) — Chowdhery et al. (Google), PaLM: Scaling Language Modeling with Pathways, arXiv 2204.02311 (2022).
• RoPE — Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv 2104.09864 (2021). The LLaMA family is a prominent adopter, not the proposer.
• Llama 1 — Touvron et al., LLaMA: Open and Efficient Foundation Language Models, arXiv 2302.13971 (2023).
• Llama 2 — Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv 2307.09288 (2023).
• Llama 3 — Grattafiori et al. (Meta), The Llama 3 Herd of Models, arXiv 2407.21783 (2024).
• Qwen2 — Yang et al. (Qwen Team), Qwen2 Technical Report, arXiv 2407.10671 (2024).
• Qwen2.5 — Qwen Team, Qwen2.5 Technical Report, arXiv 2412.15115 (2024).
• Mistral 7B (sliding-window attention) — Jiang et al., Mistral 7B, arXiv 2310.06825 (2023).
• Gemma (1) — Gemma Team (Google DeepMind), Gemma: Open Models Based on Gemini Research and Technology, arXiv 2403.08295 (2024).
• Gemma 2 (branch pre+post norm, soft-capping) — Gemma Team (Google DeepMind), Gemma 2: Improving Open Language Models at a Practical Size, arXiv 2408.00118 (2024).
• Gemma 3 (QK-Norm) — Gemma Team (Google DeepMind), Gemma 3 Technical Report, arXiv 2503.19786 (2025).
• MQA — Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, arXiv 1911.02150 (2019).
• GQA — Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv 2305.13245, EMNLP 2023.
• MLA / DeepSeek-V2 — DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, arXiv 2405.04434 (2024).
• DeepSeek-V3 — DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv 2412.19437 (2024).
• DeepSeekMoE (fine-grained + shared experts) — Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, arXiv 2401.06066 (2024), ACL 2024.
• Auxiliary-Loss-Free Balance — Wang, Gao, Zhao, Sun & Dai, Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, arXiv 2408.15664 (2024).
• Mixtral of Experts — Jiang et al., Mixtral of Experts, arXiv 2401.04088 (2024).
• Pre-LN vs Post-LN gradient argument — Xiong et al., On Layer Normalization in the Transformer Architecture, arXiv 2002.04745 (2020), ICML 2020.
• DeepNorm / DeepNet — Wang et al., DeepNet: Scaling Transformers to 1,000 Layers, arXiv 2203.00555 (2022).
• RMSNorm — Zhang & Sennrich, Root Mean Square Layer Normalization, arXiv 1910.07467 (2019), NeurIPS 2019.
• QK-Norm — Henry et al., Query-Key Normalization for Transformers, arXiv 2010.04245 (2020), EMNLP 2020 Findings.
• GELU — Hendrycks & Gimpel, Gaussian Error Linear Units (GELUs), arXiv 1606.08415 (2016).
• GLU Variants / SwiGLU — Shazeer, GLU Variants Improve Transformer, arXiv 2002.05202 (2020). Proposed and systematically evaluated these Transformer FFN variants; the original GLU and the SiLU/Swish base ideas each have earlier sources.
• Sibling tutorials in this series — attention_tutorial.md, normalization_init_tutorial.md, moe_tutorial.md, long_context_rope_yarn_mla_tutorial.md, linear_sparse_attention_tutorial.md, quantization_tutorial.md, lora_peft_tutorial.md, llm_inference_serving_tutorial.md