Llm Pretraining Pipeline Tutorial En

§0 "Input-State-Output" Master Diagram + TL;DR Cheat Sheet

Scope: this tutorial only covers decoder-only causal LMs (next-token prediction). The shifted labels, causal mask, and loss normalization in §3 all assume this setting; other pretraining objectives are out of scope. (Examples not covered: encoder-decoder, BERT-style masked LM, denoising objectives such as T5 span corruption.)

Pretraining is not "feed data, tune hyperparameters by loss" — it is a complete state machine from raw documents to checkpoints. Most interview point losses ("6ND is an exact formula", "exceeding Chinchilla means overfitting", "the causal mask naturally isolates documents") come from compressing this pipeline into one or two isolated formulas. Build the overall mental model first:

Raw web/book/code snapshots (with license records)
  → ① corpus factory (raw-document stage): extraction/encoding normalization → language identification → basic quality filtering → exact dedup → near dedup
                      → privacy/safety processing → benchmark decontamination → cluster-level data split (dedup FIRST, split SECOND!)
                      → document-level mixture quotas → deterministic sharding, emitting raw-document shards + end-to-end lineage (§2.1-§2.6)
  → ② tokenizer (a version-frozen contract; training details not repeated here — see tokenization_tutorial.md)
token stream
  → ③ tokenized stage: token-level mixture sampling (correcting the gap between document-level quotas and actual token shares, §2.7)
                      → long-document chunk/truncate (§2.9) → emit tokenized training shards (§2.8)
  → ④ data-stream/sampler layer: shard shuffle → domain-sampling RNG → with/without replacement
                      → mutually exclusive DP-rank sharding → worker/prefetch queues → cursor position (§2.10)
train token stream                                    val/test token stream (frozen version, independent scoring protocol, §2.11)
  → ⑤ objective-to-tensor: teacher-forcing shifted labels → causal / block-diagonal mask
                      → EOS/BOS boundary rules → dual padding masks → document packing (§3)
packed batch (micro-batch × seq_len)
  → ⑥ conceptual state machine of one update: model/config contract → forward → loss (Σ NLL / Σ valid target token count)
                      → backward → gradient sync (DP all-reduce; mind sum/mean reduction semantics, §4.2)
                      → unscale (AMP) / gradient clipping (once, on the final gradient after the accumulation window ends)
                      → optimizer step (AdamW etc.; formulas deferred to optimizer_lr_schedule_tutorial.md)
                      → scheduler / consumed-token counter advance → atomic checkpoint write (checkpoint boundary)
  → ⑦ lifecycle: warmup/stable/decay (LR) and late-stage data reweighting (data side; two independent things, §5.1)
                      → per-domain monitoring / spike-NaN triage (§5.4)
                      → checkpoint (weights + optimizer + RNG + data cursor + …; three strength tiers in §5.2) → resume

Step ⑥ above is the conceptual execution order of the stages within one optimizer step. The minimal implementation in §6 only covers sanity checks for the key invariants — dedup/packing/Chinchilla fitting/spike monitoring — and contains no runnable training loop or real checkpoint/resume code; the master diagram is a conceptual map for building the mental model, not a claim that §6 reproduces the whole chain.

Notation conventions:

1. $C\approx 6ND$ is only an approximation: the coefficient $6$ comes from "forward ≈ $2ND$ + backward ≈ $4ND$" (one multiply-add = 2 FLOPs), with $N$ the compute-bearing/non-embedding parameter count; it ignores the long-context attention term, vocabulary projection, norm/softmax, activation recomputation, and FLOP-counting convention differences (item-by-item analysis in §1.1); it is an algorithm-level approximate training compute, not actual GPU FLOPs, and cannot be converted directly into wall-clock time.
2. MoE keeps two separate ledgers: per-token compute is estimated with active parameters; memory/checkpoint size is counted with total parameters (details deferred to moe_tutorial.md).
3. Kaplan (2020) and Chinchilla (2022) exponents are not the same thing: Kaplan gives $N_{opt}\propto C^{0.73}, D_{opt}\propto C^{0.27}$; Chinchilla has no single set of "exact exponents" — the analytic optimum gives $N_*\propto C^{0.452}, D_*\propto C^{0.548}$ (estimation methods and fitted $\alpha,\beta$ in §1.3). The divergence is not just "more data" — training-horizon design, whether the LR schedule matches the budget, and the fitting method all differ.
4. Exceeding "20 tokens per parameter" does not mean overfitting: a fixed model fed new, non-repeated, related-distribution data usually still lowers loss; what actually happens is that the configuration is "over-trained" relative to the compute-optimal frontier point at the final compute $C=kND$ (§1.4); 20 tokens/parameter itself is an empirical approximation, not a universal constant. Deliberately over-training small models cuts deployment cost — training-compute-optimal $\ne$ lifecycle-cost-optimal.
5. The corpus factory is the largest, most self-contained block of this tutorial (§2): upstream extraction errors easily produce systematic noise, so extraction quality is often the high-leverage stage; MinHash estimates the Jaccard similarity of shingle sets, not semantic similarity; LSH only recalls candidates, which still need exact-Jaccard verification; dedup must precede splitting — near-duplicate samples must be assigned to train/val/test as whole dedup clusters, otherwise rewrites of the same content crossing splits make validation loss spuriously low.
6. Document packing's causal mask does not automatically isolate documents, and that is not a bug: many systems treat the packed token stream as a continuous EOS-separated stream, and a plain causal mask is a legitimate choice; if the semantics require document independence, you need a block-diagonal mask plus a separate boundary loss mask — blocking attention does not automatically mask cross-document prediction labels.
7. Loss normalization must be "total sum divided by total count": averaging per-micro-batch means introduces bias when valid token counts differ across batches.
8. Global batch formula: $B_{input}=B_{micro,seq}\times L_{seq}\times G_{acc}\times W_{DP}$ — only the DP world size directly multiplies independent samples; TP/PP must not be double-counted; the loss-participating $B_{target}$ is smaller, with only $L_{seq}-1$ valid targets per sequence under shifted labels (accounting details in §4.1).
9. "Deterministic resume" splits into three strength tiers: resumable training → identical sample sequence → bitwise-identical numerical trajectory; the three must not be conflated, and the states each tier requires are in §5.2.
10. Loss spike/NaN triage should follow a layered order: bad batch/data shard → LR and resume state → gradients and activations → mixed precision → cross-rank anomalies → hardware faults, keeping reproducible batch IDs/step numbers.

§1 Training budget and scaling laws

1.1　$C\approx 6ND$: where the coefficient comes from, and what it ignores

Training compute for a decoder-only dense Transformer is commonly approximated as

$$C \approx 6ND$$

The intuition: one multiply-add counts as 2 FLOPs; for each token, the forward pass runs through all the main matrix multiplies (QKV projections, attention output projection, FFN up/down projections, etc.), costing approximately $2N$ FLOPs, so the forward pass over $D$ tokens totals $\approx 2ND$. The backward pass computes both activation gradients and weight gradients, roughly twice the forward cost, i.e. $\approx 4ND$. Forward + backward $\approx 6ND$ (the $6$ is likewise an approximate coefficient; more rigorously it should be written $k\approx 6$).

Parameter-counting convention: $N$ should be read as the compute-bearing / non-embedding parameter count — the embedding lookup is an index operation, not a matrix multiply, and mechanically counting embedding parameters into $N$ systematically overestimates that portion of the compute. Whether and how the output vocabulary projection is counted separately varies across papers/implementations, so when reading any specific FLOPs estimate, first confirm its parameter-counting convention.

(This approximation was first systematically laid out by Kaplan et al. (2020); full citation in the references at the end.)

1.2　MoE: active parameters for per-token compute, total for memory

MoE keeps two ledgers: per-token compute is estimated with active parameters (the expert parameters that actually participate in that token's forward/backward computation); model capacity, memory footprint, and checkpoint file size are counted with total parameters. For a 671B/37B-style model (public configurations like DeepSeek-V3; see references at the end), the $N$ in $6ND$ uses the 37B scale to estimate the compute budget, while checkpoint storage is counted at 671B. Note that total corresponds to the aggregate model-state size; actual per-GPU memory also depends on the sharding scheme (see Q25). Routing mechanisms, load balancing, and aux-loss derivations are out of scope here; see moe_tutorial.md.

1.3　Kaplan (2020) vs Chinchilla (2022): the exponents must not be conflated

The two foundational papers give different optimal-allocation exponents; the common interview answer "the exponents differ because they lacked data" is incomplete.

Kaplan et al. (2020):

$$N_{opt}\propto C^{0.73},\qquad D_{opt}\propto C^{0.27}$$

When the compute budget grows, Kaplan's conclusion is to allocate more to parameters rather than data.

Hoffmann et al. (2022, "Chinchilla"): the paper cross-validates with multiple estimation methods and gives no single set of "exact exponents" — the different methods yield a family of close but not identical results, roughly 0.50/0.50, 0.49/0.51, and 0.46/0.54. The most frequently cited parametric loss model is written as

$$L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

with fitted values around $\alpha\approx 0.34,\ \beta\approx 0.28$. $E$ is the fitted asymptotic loss floor, roughly comprising the entropy of the data itself plus the approximation error of the model family — though the fit cannot uniquely decompose the two. $A/N^\alpha$ is the price paid for "parameters not large enough", $B/D^\beta$ the price for "data not plentiful enough".

Constrained optimization of $L(N,D)$ under the compute budget $C=kND$ ($k=6$, see §1.1), eliminating variables via Lagrange multipliers, yields the analytic solution:

$$N_* = \left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha+\beta}}\left(\frac{C}{k}\right)^{\frac{\beta}{\alpha+\beta}},\qquad D_* = \left(\frac{\beta B}{\alpha A}\right)^{\frac{1}{\alpha+\beta}}\left(\frac{C}{k}\right)^{\frac{\alpha}{\alpha+\beta}}$$

Substituting the rounded $\alpha=0.34,\beta=0.28$: $\dfrac{\beta}{\alpha+\beta}=\dfrac{0.28}{0.62}\approx 0.452$, $\dfrac{\alpha}{\alpha+\beta}=\dfrac{0.34}{0.62}\approx 0.548$, i.e.

$$N_*\propto C^{0.452},\qquad D_*\propto C^{0.548}$$

(Computed with the paper's full-precision $\alpha,\beta$, the commonly reported values are about 0.46/0.54; [D06] in §6 turns these exponents into an executable fitting unit test.)

1.4　"Exceeding the Chinchilla ratio = overfitting" is a false statement

This is a common misconception that must be corrected. Overfitting is defined as training loss falling while validation loss rises — a model of fixed size $N$ fed additional new (non-repeated, same- or related-distribution) data usually still lowers loss, which does not meet that definition. What actually happens: as $D$ keeps growing at fixed $N$, the final compute $C=kND$ (§1.1) grows in lockstep — you arrive at a larger $C$, and relative to that $C$'s compute-optimal frontier point $(N_*,D_*)$ (§1.3), the configuration $(N,D)$ has a smaller $N$ and a larger $D$. This configuration is called "over-trained" — an economic judgment relative to the compute-optimal frontier, not a training error, and certainly not overfitting.

Industry deliberately trains many small models far past the Chinchilla-optimal ratio (e.g. choosing a smaller model at equal capability and feeding it more tokens), justified by deployment-side inference cost (memory, latency, per-token serving cost) — trading one-off extra training compute for cheaper inference over the model's entire lifecycle. This is a trade-off between "training-compute-optimal" and "lifecycle-cost-optimal", not a "wrong recipe".

1.5　Choosing $N,D$ for a given budget: an engineering checklist

1. First pin down the optimization objective: training-compute-optimal (use the $N_*,D_*$ analytic solution directly) or inference-aware optimal (first cap a smaller $N$ by the deployment budget; at fixed $C$, $D$ grows accordingly — i.e. deliberately "over-train" a small model);
2. Long-context training brings in the $L_{\text{seq}}^2$ attention term that $6ND$ ignores (§1.1); budget for it separately rather than applying $6ND$ across the board;
3. MoE models use active parameters for the compute budget (§1.2) and total parameters when reporting storage/memory budgets;
4. Clarify the accounting for $D$ — the number of tokens "presented during training", or the deduplicated "effective" data volume (§2.7); a highly repetitive corpus makes the "effective data volume" fall below the nominal $D$, which is also the concern of the post-Chinchilla line of "data-constrained scaling" work.

§2 The corpus factory: from web snapshots to training shards

This section is the largest, most self-contained block of the tutorial — whether pretraining "runs stably and correctly" is often decided more by corpus-pipeline design than by model architecture or hyperparameter choices.

2.1　The eleven-step pipeline overview + lineage

An auditable, recoverable data pipeline roughly comprises the following stages (order is critical, especially "dedup before split"):

Two stages, not one line:

1. Raw-document stage: the 11 steps above — step 10's "mixture sampling" is a document-level quota (what fraction of documents each domain contributes), and step 11 emits raw-document shards;
2. The tokenizer runs after this: turning raw-document shards into a token stream (interface contract in §2.8);
3. Tokenized stage: token-level mixture sampling (the token-level shares implied by document-level quotas need not match the configuration, §2.7) → long-document chunk/truncate (§2.9) → emit tokenized training shards → the data-stream/sampler layer of §2.10; after the split, train and val/test diverge (§2.11).

Mixture sampling happens once at the document level and once at the token level, with different meanings — reading it as a single step in the table is the easiest way to misread the order of this pipeline.

Data lineage: every training sample should ideally be traceable to — source dataset/URL, snapshot/crawl version, extraction and filtering pipeline version, quality filters and thresholds it hit, its dedup cluster ID, split assignment, and the final shard file plus offset it landed in. These records are the only basis, months later, for investigating "did some output memorize training data" or "was some benchmark contaminated".

2.2　Extraction quality: often the high-leverage stage, not "rigor matters more"

The most upstream steps — web main-content extraction (stripping nav bars, ads, footer templates, comment-section noise) and encoding/Unicode normalization — are high-leverage stages: upstream extraction errors systematically press large amounts of structural noise (navigation text, repeated templates, mojibake) into every document, and their impact often exceeds fine-tuning of downstream quality classifiers — this is an empirical observation, not a universal ordering that "extraction always outranks classifiers" (indirectly supported by the C4 corpus audit of Dodge et al. and the RefinedWeb/FineWeb experience of Penedo et al.). The training corpus's normalization rules must match those used when the tokenizer was trained, otherwise "online/offline segmentation mismatch" drift appears — tokenizer training details, normalization options, and byte fallback are in tokenization_tutorial.md §3–§4.

Language identification usually routes documents with a lightweight classifier (e.g. a fastText-style n-gram model); low-confidence judgments on code-switched text and short text are common engineering edge cases. Basic quality filtering typically stacks several heuristics (line-length distribution, symbol/letter ratio, repeated-n-gram share) and/or a small auxiliary LM's perplexity score — in essence "use cheap signals to screen out the worst batch", not fine-grained semantic judgment.

2.3　Exact dedup vs near dedup

Exact dedup: byte-level/hash-level full matching (whole-document hash dedup, or suffix-array-based cross-document duplicate-substring dedup); it catches only identical content, missing near-duplicates like "one word changed in the title".

Near dedup (MinHash + LSH): represent documents as shingle (n-gram) sets, estimate pairwise Jaccard similarity with MinHash (Broder, 1997; see references at the end), then use LSH bucketing for candidate recall. Three boundaries that must be clarified:

• LSH banding only does candidate recall: candidate pairs usually still need exact-Jaccard verification, because LSH bucketing only guarantees that candidate probability rises monotonically with similarity (under the idealized assumption that MinHash rows/hash functions are approximately independent, $P=1-(1-s^r)^b$, with $b$ bands of $r$ rows each); it does not guarantee a candidate is a true near-duplicate — treating candidates as conclusions introduces false positives.
• Semantic dedup is not an unconditional upgrade of near dedup: embedding-threshold semantic dedup can wrongly delete legitimate "same topic, different expression" text, and this over-deletion hits low-resource languages, dialects, and templated technical documents (whose embedding distributions are more concentrated) unevenly — it must not be swapped in as a "fancier MinHash"; the two address different levels of duplication.

2.4　Privacy/safety processing + benchmark decontamination

Privacy/safety processing (PII redaction — emails, phone numbers, keys; illegal and harmful content filtering) is a process independent of "quality level"; it usually needs dedicated detection rules and thresholds and should not be tuned together with quality filters.

Benchmark decontamination: detect n-gram overlap between the training corpus and target eval sets (e.g. common QA/reasoning benchmarks), removing or flagging overlapping documents. Exact matching at the 13-gram scale is the concrete practice explicitly adopted in the GPT-3 paper (Brown et al., 2020; see references at the end) to detect overlap between training corpus and eval sets — a widely borrowed specific precedent, but teams differ in the n-gram lengths they actually use, so it should not be treated as an industry-wide standard. This guards against benchmark contamination ("seeing the eval answers during training") and is a completely different problem from the train-val leakage of §2.6 below — do not conflate them.

2.5　Dedup before split: cluster-level split (critical)

If you split first and dedup later: near-duplicate content (mirror sites, reposted/rewritten articles, different instances of templated documents) can easily land one copy in train and another in val — the model has "seen" a near-verbatim copy of a validation sample during training, so validation loss is spuriously low, not because the model generalizes well but because it has effectively memorized a rewrite of the validation sample. This misleads early stopping, model selection, and any downstream quality reporting based on that validation loss (for a systematic discussion of this phenomenon and dedup's effect on downstream model quality, see Lee et al., 2021, in the references).

Correct order: first run near dedup, partitioning all documents into connected components (dedup clusters); then assign whole clusters atomically to train/val/test — documents in the same cluster must never be split across splits. The key is the transitive closure: even if document A was only directly compared with B, and B only with C (A and C never directly compared), as long as A~B and B~C, A and C must land in the same split — [D01] in §6 turns this rule into an executable assertion with union-find.

2.6　Three kinds of "leakage" that must be discussed separately

A more accurate name for the third is "cross-sample context coupling": the later document reads earlier tokens, not future labels — a modeling choice at the level of training dynamics/statistical independence, not information leakage in the evaluation sense. Whether to sever it with a block-diagonal mask depends on whether you require the modeling assumption "every document must be a statistically independent sample"; see §3.4.

2.7　Mixture sampling: domain weights need not equal token weights

Mixture sampling has at least three different "sampling units", and the same "domain weight" configuration produces different final token distributions under each:

• Sample by document: first select a document pool by domain weights, then draw documents uniformly/weighted from the pool, taking whole documents;
• Sample by token: cut fixed-length blocks directly from the concatenated token stream, decoupled from document boundaries;
• Sample domain first, then document: first pick a domain by domain weights, then pick a document within it.

The differences come from domains having different average document lengths — a domain with longer average documents contributes more tokens at the same document-count weight. The relationship between "domain weights" and token weights splits into two cases:

• Configured as document counts/document weights but used directly as token weights: the two are usually unequal — this is the source of the bias;
• The configuration itself is defined as token-level target shares: in expectation the two can be equal.

Whichever sampling unit is used, the "actual token share" must be measured and reported separately after sampling, not just the configured values. For a systematic approach to domain reweighting, see DoReMi (Xie et al., 2023, references at the end).

This also ties into the accounting for $D$ in §1: the $D$ in scaling laws usually means the tokens presented to the model during training (total tokens flowing through the optimizer), and is not guaranteed to equal the corpus's unique token total — if some domain is repeatedly oversampled (multiple epochs), or the corpus still contains many residual near-duplicates, the nominal $D$ is inflated above the "effective" data volume, and the excess does not carry the same statistical information as new data; for a systematic discussion of how repeated data affects effective $D$ in such "data-constrained" settings, see Muennighoff et al. (2023, references at the end) — only the qualitative conclusion is stated here, without adopting their specific numbers.

2.8　Deterministic sharding + the tokenizer interface contract

Deterministic sharding: given the same corpus + the same mixture configuration + the same random seed, "which document goes into which shard file, at which position" must be reproducible — this directly underpins the deterministic resume of §5.2 (after a restart you must be able to pinpoint exactly where consumption stopped) and the spike triage of §5.4 ("reproduce the specific batch that caused this spike" depends on exactly reconstructing the data consumed at the time).

Tokenizer interface contract (a pointer; training is not re-taught here): the corpus factory's output interface toward the tokenizer should be a small, frozen contract — a version-locked tokenizer artifact (with hash), fixed BOS/EOS/PAD IDs, fixed normalization rules, byte-fallback behavior, and a bytes/token ratio for budget estimation. How the tokenizer itself is trained (BPE/WordPiece/Unigram) and SentencePiece framework details are in tokenization_tutorial.md §2–§5; this tutorial only consumes the contract and does not re-derive it.

2.9　Long-document chunking / truncation

A single document's raw length often exceeds the training context length (sometimes by many multiples — a book or a long code repository can far exceed a window of a few thousand to tens of thousands of tokens). Such documents must be chunked before entering packing, with several decision points that must be made explicitly:

• Whether to overlap: non-overlapping chunking is simplest but hard-cuts the context at every chunk boundary; overlapping chunking (e.g. keeping an overlap region of tens to hundreds of tokens between adjacent chunks) mitigates the missing context at boundaries, at the cost of the same original text being shared by multiple chunks;
• Whether the overlap region counts toward loss twice: if two adjacent chunks share a stretch of overlapping tokens, you must decide whether that content contributes target loss in only one chunk or in both — counting both effectively "oversamples" that content, distorting its true weight in the total $D$ (§1, and $B_{target}$ of §4.1); usually the loss should be kept only in the chunk that sees the longer preceding context, marking it as pure context (no loss) in the other;
• Whether to insert EOS at artificial chunk boundaries: this is a semantic choice, not a standard answer — if each chunk should be treated as an independent sample (echoing the packing semantics of §3.4), insert EOS/BOS boundaries at the cuts and assign new doc-ids, so the downstream block-diagonal mask and boundary loss-mask rules correctly recognize "these are actually two artificially cut segments of the same document"; without boundary markers, multiple chunks are treated as "continuous continuation of the same document";
• How position ids / doc-ids carry over: if you choose "each chunk is an independent sample", each chunk should have its own doc-id (and decide whether to reset position ids per the discussion in §3.4); if you choose "chunks are just cuts, logically still the continuation of one document", the doc-id should stay consistent across chunks and position ids should be numbered continuously rather than restarting from 0 per chunk — this choice must stay consistent with how a "document" is defined at training/inference time, otherwise you get inconsistencies like "treated as independent samples in training but assumed positionally continuous at inference".

None of these decisions has a universal default, but they must be recorded explicitly in the lineage (§2.1): if a piece of training data has been chunked, its lineage should trace back to "which chunk of which original document, whether it overlaps with adjacent chunks, whether the overlap region counts toward loss" — otherwise it is impossible to determine afterward whether an observed problem (say, anomalous loss weight on some content) was caused by the chunking policy.

2.10　The data-stream / sampler layer

After tokenized training shards are written to disk, what actually reads them at training time is a data-stream / sampler layer that is usually skipped over, yet whose state definitions §4 (gradient accumulation) and §5.2 (deterministic resume) repeatedly depend on:

• Shard shuffle: whether the shard-file read order is shuffled before each epoch (or each traversal), and the random seed/state used for the shuffle;
• Domain-sampling RNG: if the mixture is sampled dynamically at training time (rather than pre-mixed offline with fixed ratios), you need a domain-selection random number generator independent of other randomness sources (dropout, data shuffling), whose state must also be capturable by checkpoints;
• With / without replacement: whether documents/shards within a domain may be resampled — multi-epoch training, or data insufficient to support the compute-optimal $D$ (§1.5), usually requires sampling with replacement, and the "repetition" itself must then be recorded, because it directly affects the "nominal $D$ vs effective data volume" accounting discussed in §2.7;
• Mutually exclusive DP-rank sharding: the shard/sample sets read by each DP rank must be pairwise disjoint, otherwise the same data is counted by multiple ranks simultaneously, implicitly double-counting $D$ (this constraint also appears in §4.4);
• Worker/prefetch state: multi-process/multi-thread dataloaders typically prefetch several batches into a queue, and the queue's own state (what has been prefetched, what remains unconsumed) is also part of the training state — if resume restores only the "which sample have I read" cursor without correctly handling data already dequeued but not yet consumed, that data may be skipped or consumed twice;
• Cursor position: all the randomness and ordering decisions above must ultimately converge to a "cursor" that can be saved and restored precisely — usually "which byte/sample offset in which shard file"; the precision of this cursor determines how exact the "identical sample sequence" tier of §5.2 can be.

When deterministic resume (§5.2) is discussed, this layer's state is often waved through as "the data iterator position", but a real implementation involves at least the five independent state classes above, and failing to save any one of them degrades the "identical sample sequence" promise of resume.

2.11　After the cluster split: where train and val/test diverge

The cluster-level split of §2.5 settles only one thing — "which dedup clusters belong to which split"; once the split is done, train and val/test follow two completely different downstream processes and must not be vaguely handled by "reusing the same training mixture":

• train: continues through the full §2.7-§2.10 process — mixture sampling (possibly with dynamically adjusted domain weights, echoing the late-stage data reweighting of §5.1), shuffling, with-replacement resampling (if needed), and all the state of the data-stream/sampler layer;
• val/test: must be version-frozen — once fixed, the same val/test set should keep its sample content, size, and scoring protocol unchanged for the entire training lifecycle (and across multiple training runs when comparing fairly); it should not be affected by the dynamically adjusted mixture like train, nor "diluted" or "reweighted" by resampling — per-domain reporting (§5.3) requires an independent, comparable validation set for each domain, not a subset that tracks the train mixture;
• if val/test also needs to cover multiple domains, the within-domain sampling should be determined once at the split stage and frozen, rather than dynamically resampled each epoch/run like train — otherwise validation losses across checkpoints are no longer measured with the same yardstick.

Casually treating val/test as "a scaled-down version of the train mixture" is the most easily overlooked stage of this pipeline, and it directly contaminates the per-domain monitoring conclusions of §5.3 and the checkpoint-selection conclusions of §5.5.

§3 Objective-to-tensor: how the token stream becomes supervision

3.1　Teacher forcing + shifted labels

Standard next-token training uses teacher forcing: given a token sequence $x_1,\dots,x_T$, the label at position $i$ is $x_{i+1}$ (the sequence shifted right by one); the model outputs the predictive distribution $p(x_{i+1}\mid x_{\le i})$ at position $i$, and the loss is the NLL over all valid target positions (§3.5 gives the exact normalization formula). The last position has no next token, so its label is marked IGNORE (this tutorial follows the PyTorch convention and uses the sentinel value $-100$).

3.2　Causal mask + EOS/BOS boundary rules

A plain causal mask $M[q,k]=\mathbb{1}[k\le q]$ only guarantees "cannot see the future". Around document boundaries there is one supervision rule that is frequently written incorrectly:

• Keep the supervision "last content token of a document → EOS" — this is the direct signal from which the model learns "when to stop";
• Mask (per configuration) the supervision "previous document's EOS → next document's first token (or BOS)" — the "label" at that position is merely an artifact of concatenation, not any meaningful generation continuation.

Both rules can be produced automatically by one and the same document-boundary rule: treat EOS as the last token of its document (same id as the rest of that document in the doc-id array), and construct labels with the single rule "a label is valid iff the current position and the next position belong to the same document" — content-end token to EOS naturally shares a document (rule says valid), while EOS to the next document's first token naturally crosses documents (rule says invalid). No separate special-case logic for EOS is needed; [D04] in §6 demonstrates this.

One pointer worth a sentence: when assembling chat templates, using both the template's built-in BOS and the tokenizer's add_special_tokens easily inserts BOS twice — a tokenizer/inference-side correctness issue detailed in tokenization_tutorial.md §5.5; this section only notes that it pollutes the supervision signal and does not expand on it.

3.3　Padding: the loss mask is mandatory; whether the key mask is depends on layout

Padding involves two distinct concerns:

1. Key padding mask: prevent query positions from attending to padding keys;
2. Loss mask: exclude positions whose target is padding from the loss (otherwise gradients are computed against meaningless filler targets).
3. The loss mask is always mandatory — without it, the loss is guaranteed to be polluted by filler targets;
4. The key mask can be omitted under strict causal + right padding — real query positions are naturally smaller than the padding start, so the causal mask ($k\le q$) already blocks those future padding keys, and omitting it does not change real tokens' outputs;
5. The key mask is mandatory under left padding, holes inside sequences, or other layouts — the causal condition alone cannot exclude padding keys, and real queries can easily read meaningless content.

Recommendation: a general-purpose implementation constructs both; only omit the key mask, with justification, when you can confirm you are strictly in the "right padding + causal" special case and are willing to maintain that implicit assumption long-term.

3.4　Document packing: cross-document attention is not a bug

Packing multiple documents into one training sequence is a common way to raise effective batch utilization. The core facts:

• A plain causal mask only prevents "reading the future"; it does not automatically prevent a later document from reading an earlier document's history tokens — this is not an implementation defect: many systems deliberately treat the packed token stream as one EOS-separated continuous stream with a plain causal mask (as §2.6 explained, a sequence-semantics choice, not data leakage);
• If the semantics require "each document is an independent sample", you need a block-diagonal causal mask:

$$M[q,k] = (k\le q)\ \wedge\ (\text{docid}[q]=\text{docid}[k])$$

Whether to reset position IDs within each document depends on the position-encoding scheme and training semantics — it is not a universal rule:

• Absolute position embeddings: without a reset, later documents start from a huge position number that does not belong to them; usually reset;
• Standard RoPE: depends only on the relative position difference between query and key — once the block-diagonal mask restricts attention to within documents, adding the same offset to a whole document does not, in ideal math, change the relative phases within the document, so attention is not polluted merely because "numbering did not start from 0";
• Variants and engineering considerations: RoPE length-extrapolation/scaling variants, numerical behavior at finite precision, and the desire to train every document strictly from position 0 (aligning with the position distribution at inference) — these still lead most implementations to reset;

• Blocking attention does not automatically mask cross-document prediction labels — the most common bug in packing implementations: a team adds the block-diagonal mask and assumes "documents are now fully independent", but without separate label handling, document A's last position is still, by default, asked to predict document B's first token (a cross-document target through and through), which must be excluded separately by the boundary loss mask of §3.2.

Numerical example (tokens = [11,12,13,21,22], doc IDs = [0,0,0,1,1]): under independent-document semantics the shifted labels are [12,13,IGNORE,22,IGNORE]; the block mask asserts M[3,2]=False (position 3 belongs to document 1, position 2 to document 0 — blocked for crossing documents even though $k\le q$), M[4,0]=False, M[4,3]=True; under continuous-stream semantics the same position has M[3,2]=True — a legitimate configuration choice, not an implementation failure.

3.5　Loss normalization: total sum over total count, not a mean of means

The correct training loss is

$$\mathcal{L} = \frac{\sum_i \mathrm{NLL}_i}{\#\{\text{valid target tokens}\}}$$

Sum the NLL over all valid target tokens in the global batch, then divide by the total valid-token count. A common but insidious bug: first compute "the mean loss within this micro-batch" for each micro-batch, then average those means (e.g. across the micro-batches of gradient accumulation, or across DP ranks). When micro-batches have different valid-token counts (different padding ratios, different packing efficiency, or an in-progress short-to-long context-length switch), this "mean of means" implicitly overweights micro-batches with few tokens and underweights those with many, deviating from the truly token-weighted loss. A correct implementation must accumulate two quantities — "NLL sum" and "valid token count" — across the whole gradient-accumulation window (and across DP-rank communication), dividing once at the end.

If the model is MoE, the total loss usually also adds the router's auxiliary loss term per configuration (router aux loss, typically far smaller in magnitude than the main loss) — its mechanics are out of scope here; see moe_tutorial.md.

3.6　Exact PPL/BPB formulas

$$\mathrm{PPL} = \exp\left(\frac{\sum_i \mathrm{NLL}_i}{\#\text{valid target tokens}}\right),\qquad \mathrm{BPB} = \frac{\sum_i \mathrm{NLL}_i}{\#\text{bytes}\cdot\ln 2}$$

PPL is directly comparable only when tokenizer, normalization, and evaluation text are all identical — the denominator (token count) itself depends on the tokenizer, so even at identical total NLL, a tokenizer that segments more finely drives loss/token down. BPB replaces the denominator with the UTF-8 byte count of the same text, making it more robust to tokenizer differences, though still affected by whether encoding/preprocessing match. The full derivation and the numerical example of "loss/token differs by 2x yet BPB is identical" are in tokenization_tutorial.md §1.3 and §7 [D07]; this section does not repeat them and only stresses the exact form of these two formulas.

§4 Update accounting: what the global batch actually is

4.1　Micro-batch / DP / gradient accumulation: the global batch formula, and three different token-counting conventions

Pin the notation first: $B_{micro,seq}$ is the number of sequences per micro-batch per DP replica, $L_{seq}$ the sequence length, $G_{acc}$ the gradient-accumulation steps, $W_{DP}$ the data-parallel world size.

A DP replica is not a GPU: when TP/PP > 1, one DP replica itself spans multiple GPUs — "per GPU" and "per DP replica" are not the same thing.

Tokens fed to the model (processed token slots, regardless of valid loss participation):

$$B_{input} = B_{micro,seq}\times L_{seq}\times G_{acc}\times W_{DP}$$

Only $W_{DP}$ directly multiplies the number of independent samples. Tensor parallelism (TP) and pipeline parallelism (PP) partition the computation of the same samples; multiplying their world sizes into $B_{input}$ is the most common source of global-batch miscalculation. (Communication primitives and partitioning mechanics are deferred to distributed_training_tutorial.md.)

Valid targets actually participating in the loss:

$$B_{target} = \sum_i m_i$$

where $m_i$ is the valid-target mask count of sequence $i$ (the loss-mask sum of §3.5). $B_{input}$ and $B_{target}$ are generally unequal, even with no padding at all: under shifted labels, each sequence's last position has no next token and is marked IGNORE (§3.1), so valid targets per sequence number $L_{seq}-1$, not $L_{seq}$; stacking on padding (§3.3) or packing's document-boundary loss mask (§3.2, §3.4) shrinks $B_{target}$ further below $B_{input}$.

Which quantity each of the three accounting conventions maps to — do not mix them:

Once the three are mixed, the question "how many tokens did this training run feed" produces several mutually inconsistent numbers in reports.

4.2　Conditions for gradient accumulation to be strictly equivalent to "one big batch"

For gradient accumulation to be strictly equivalent to running one big batch directly, all of the following must hold:

1. Same samples — the data consumed within the accumulation window must exactly match what the big batch would consume at once; no samples missed or repeated due to different sharding;
2. Normalize over all valid tokens (§3.5) — not a mean over per-micro-batch means;
3. Exactly one optimizer step — gradients are only summed/averaged within the accumulation window; the weight update happens once, at the window's end;
4. Consistent final-gradient post-processing — gradient clipping must act on the final gradient after accumulation completes, not clip each micro-batch's local gradient separately;
5. Consistent randomness and batch-dependent state — dropout and other randomness, plus any cross-sample running statistics, must behave as if computed once on the full big batch;

6. Match the communication library's reduction semantics — the final invariant: the global gradient should equal $\nabla\left(\sum_r \mathrm{NLL}_r \big/ \sum_r M_r\right)$ ($r$ ranging over DP ranks, $M_r$ that rank's valid-token count).

   • Frameworks like DDP/FSDP average multi-rank gradients by default rather than summing: if the local loss was already normalized as "NLL sum / this rank's valid tokens", the default averaging implicitly divides by the world size once more;
   • Pick one fix: explicitly switch to sum-reduce and normalize by the global valid-token count, or pre-multiply by $W_{DP}$ when normalizing locally.
   • The one core check: the local loss and the communication layer must not both divide by $W_{DP}$.

The points above guarantee mathematical equivalence — under exact real arithmetic, the accumulated gradient and the one-shot big-batch gradient are the same quantity; they do not guarantee bitwise-identical floating-point values: accumulation order, the all-reduce reduction-tree structure, etc. still cause differences in the last few significant digits (a distinction consistent with the three-tier deterministic-resume framework of §5.2).

4.3　Numerical precision: operational constraints only, no number-format textbook

BF16 has the same exponent width as FP32 (similar dynamic range) and in pretraining typically overflows less readily than FP16; FP16's narrower dynamic range makes it more dependent on loss scaling to keep small gradients from underflowing to zero. Only the operational constraints relevant to the spike/NaN triage of §5.4 are given here; floating-point format derivations are not expanded.

4.4　Sibling boundaries (strictly enforced)

• optimizer_lr_schedule_tutorial.md: owns the AdamW update derivation and warmup/cosine/WSD scheduling math; this tutorial cares only about why optimizer state must be saved with checkpoints (§5.2), whether the schedule advances by step or by token count, the coupling when batch size changes, and the lifecycle roles of the three phases (§5.1) — all specific formulas are deferred to that tutorial.
• distributed_training_tutorial.md: owns the communication primitives and partitioning mechanics of ZeRO/FSDP/TP/PP/CP; this tutorial cares only about the global-batch accounting of §4.1 and that the data consumed by DP ranks must not overlap (otherwise $D$ is double-counted).
• Uncovered boundary: how checkpoints reshard when the world size changes, and whether the resharded run can match the original trajectory — neither this tutorial nor that sibling gives a systematic answer; this tutorial only states the weaker qualitative conclusion in §5.2 that "changing the world size usually cannot guarantee a bitwise-identical trajectory".

§5 Lifecycle: from warmup to deterministic resume

5.1　Warmup/stable/decay/cooldown + data-phase changes

The lifecycle divides roughly into phases along the LR schedule (only the roles of the phases are covered here; the math is in optimizer_lr_schedule_tutorial.md): warmup suppresses early-training instability; stable/plateau occupies the bulk of training. In the final stretch, two things often happen simultaneously, but they are controlled by different configurations:

• Decay/cooldown is an LR-side concept — the learning rate steps down to a very small value per the schedule; formulas and schedule shapes are deferred to optimizer_lr_schedule_tutorial.md;
• Data annealing/late-stage reweighting is an independent data-side concept — raising the weight of high-quality/curated data in the mixture near the end of training (described in public training reports such as Llama 3; see references at the end).

A training recipe can perfectly well do LR decay without adjusting data weights, and vice versa — do not treat "the LR is decaying" and "the data is being reweighted" as two phrasings of the same thing.

Another lifecycle axis is the context-length curriculum: most of the training budget uses shorter sequences, switching to long context late. This changes $L_{seq}$, meaning the global-batch accounting of §4.1 and the consumed-token counter must both be recomputed consistently at the switch point rather than silently carrying over pre-switch assumptions.

5.2　"Deterministic resume" must be split into three promises of different strength

The sentence "after resume, the training trajectory is exactly as if uninterrupted" is itself ambiguous — different engineering goals need very different necessary conditions, best discussed as three increasing strength tiers:

1. Resumable training (weakest): the model can restore weights and optimizer state from the checkpoint and continue lowering loss, with no requirement that data order or numerics align exactly. Only model weights, optimizer state, and the LR schedule/consumed-token count are needed.
2. Identical sample sequence: after restart, the data order the model sees (which sample at which step) exactly matches the uninterrupted run, without requiring bitwise-identical numerics. Beyond tier 1, this additionally requires: RNG states (the generators in Python/framework/CUDA etc.), the data-iterator position, the shuffle epoch, the sampler state, and the dataloader worker/prefetch queue state (§2.10; otherwise multi-worker prefetch order is itself a nondeterminism source); multi-rank training also needs per-rank checkpoint metadata.
3. Numerically/bitwise identical trajectory (strongest, usually not fully attainable): on top of tier 2, this needs the AMP loss-scaler state (otherwise the dynamic loss scale restarts from a different point and the gradient-scaling path forks within the first few steps), any "in-flight" pending gradients inside an unfinished accumulation window (avoidable if checkpoints are only taken at accumulation boundaries), deterministic kernels (nondeterministic cuDNN/cuBLAS operators must be explicitly disabled), and exactly identical software versions and communication topology (different GPU counts/collective implementations reduce floats in different orders, accumulating ULP-level differences). When the world size changes, even the most careful resharding semantics usually cannot guarantee a trajectory bitwise-identical to the unchanged-world-size run — the data partitioning and the gradient-reduction tree structure both change.

Missing the state required by any tier degrades the promise to a weaker tier — checkpoint metadata should explicitly declare which tier it guarantees (the world-size-change resharding boundary is in §4.4).

5.3　Monitoring: per-domain validation loss

Validation loss should be reported per domain (code/math/low-resource languages/long context, etc.), not just as one aggregate loss — the aggregate is easily dominated by the domain with the highest token weight (e.g. with web text at 80% of the mixture, the aggregate loss curve basically reflects only web-text quality), masking the degradation of a lower-weight but strategically important capability (code correctness, mathematical reasoning, some low-resource language, long-context retrieval) until it is far too late to fix cheaply.

5.4　Loss spike / NaN: layered localization

The recommended triage order (not guessing causes all at once, but descending the "cheapest checks first" hierarchy):

1. Bad batch / data shard — first see whether the spike reproduces in isolation on the same batch/shard (this depends on the deterministic sharding of §2.8, the sampler/cursor state of §2.10, and lineage; otherwise you cannot even locate "which batch");
2. LR and resume state — a misconfigured LR schedule, or a schedule counter accidentally reset after a resume, can both produce anomalous updates whose actual step count differs from expectations;
3. Gradients and activations — anomalous gradient norms, exploding activation norms, runaway logit magnitude in some layer; the runaway mechanisms are deferred to normalization_init_tutorial.md (that sibling's boundary is described in §5.6); this section only lists these quantities as triage items;
4. Mixed precision — BF16/FP16 overflow/underflow (§4.3);
5. Cross-rank anomalies — one DP/TP/PP rank quietly diverging (e.g. a GPU with hardware trouble) while the others look normal on the surface; only per-rank loss/gradient statistics reveal this;
6. Hardware faults — ECC errors, single-GPU failures, collective-communication jitter.

Keeping reproducible batch IDs / step numbers / shard attribution (§2.8, §2.10) is what enables after-the-fact replay and localization instead of guessing from memory. [D07] in §6 is the automated first line of defense — a rolling median/MAD detector responsible for flagging "when to start the triage above", not for judging root cause.

5.5　Checkpoint selection

Checkpoint selection for downstream deployment should rely on the per-domain validation sets of §5.3 / a dedicated eval-set combination, not just the training loss — under a fixed data mixture, training loss mainly reflects fitting progress and does not directly correspond to generalization or the specific capability profile you actually care about.

5.6　Sibling boundaries

• normalization_init_tutorial.md owns the full formula derivations for norm/residual/initialization and the stability mechanisms themselves; this tutorial only cites the names of the diagnostic quantities — activation norm / gradient norm / logit magnitude / residual instability — in NaN diagnosis (§5.4) without re-deriving them. That sibling explains "why these mechanisms make training more stable"; it is not a copy-paste monitoring-metrics runbook, and concrete monitoring thresholds still require your own engineering.
• moe_tutorial.md owns the full mechanics of routing and load balancing; this tutorial only cares about the router aux-loss term that may appear in the total loss (§3.5) and the two ledger rules — active parameters for compute budget, total parameters for storage/memory (§1.2). Routing mechanics are uniformly deferred to that tutorial.

§6 From-scratch implementation: corpus → packing → key-invariant sanity checks (not a full training loop)

The complete runnable script is code/pretraining_pipeline.py (pure Python standard library, zero third-party dependencies; all [D01]–[D07] sanity checks finish in seconds on CPU). Four demo groups chain into a mini pipeline: text → dedup → pack + loss mask → synthetic budget planner → spike monitor — this covers the key invariants that are the easiest to get wrong and the most worth backstopping with executable assertions, not an end-to-end runnable training loop: the script has no real forward/backward/optimizer step, and no real checkpoint writing or resume (for that conceptual state machine, see the master diagram in §0 and the three-tier framework of §5.2). The main text shows only the three most essential code segments; the full document-packing functional test, EOS boundary (§3.2), and padding (§3.3) are folded into the same script as short test functions and not pasted line by line.

[D01] MinHash / LSH / cluster-split transitive closure (executable version of §2.3, §2.5):

doc_a = set(range(1, 11))                 # {1,...,10}
doc_b = set(range(1, 9)) | {11, 12}        # {1,...,8,11,12}
j_exact = exact_jaccard(doc_a, doc_b)
assert abs(j_exact - 2 / 3) < 1e-12        # |A∩B|=8, |A∪B|=12 -> 8/12

sig_a = minhash_signature(doc_a, num_perm=1024, seed=12345)
sig_b = minhash_signature(doc_b, num_perm=1024, seed=12345)
j_est = minhash_estimate(sig_a, sig_b)     # MinHash ESTIMATE, not exact
tol = 6 * math.sqrt((2 / 3) * (1 / 3) / 1024)   # principled sampling tolerance
assert abs(j_est - 2 / 3) < tol             # close, not asserted exactly equal

p_high = lsh_candidate_probability(2 / 3, b=128, r=8)   # P = 1-(1-s^r)^b
p_low = lsh_candidate_probability(0.05, b=128, r=8)
assert p_high > 0.9 and p_low < 0.1         # high-similarity candidates recalled far more often

uf = UnionFind(4)               # docs: 0=A, 1=B, 2=C, 3=D(unrelated)
uf.union(0, 1); uf.union(1, 2)  # A~B found, B~C found (A,C never directly compared)
assert uf.find(0) == uf.find(2) # transitive closure still merges them into one cluster
# -> every doc in this cluster MUST receive the same train/val/test split label

[D02]/[D03] document packing: shifted labels + block-diagonal mask + functional test (executable version of §3.4 — the most essential and most error-prone segment of the whole codebase):

tokens = [11, 12, 13, 21, 22]
doc_ids = [0, 0, 0, 1, 1]

labels_indep = shifted_labels_independent_docs(tokens, doc_ids)
assert labels_indep == [12, 13, IGNORE, 22, IGNORE]      # doc boundary at index 2 masked
labels_cont = shifted_labels_continuous(tokens)
assert labels_cont == [12, 13, 21, 22, IGNORE]           # continuous stream: no boundary masking

m_block = block_diagonal_causal_mask(doc_ids)
m_cont = continuous_causal_mask(len(tokens))
assert m_block[3][2] is False and m_block[4][0] is False and m_block[4][3] is True
assert m_cont[3][2] is True          # SAME (q,k); continuous mode allows it -- a config choice

values = [100, 100, 100, 0, 0]       # doc0=100s, doc1=0s
assert masked_uniform_attention_output(m_cont[3], values) == 75.0    # (100+100+100+0)/4
assert masked_uniform_attention_output(m_block[3], values) == 0.0    # only key=3 allowed

values2 = [1000, 1000, 1000, 0, 0]   # bump doc0's values 10x
assert masked_uniform_attention_output(m_block[3], values2) == 0.0   # doc1's output: UNCHANGED
assert masked_uniform_attention_output(m_cont[3], values2) == 750.0  # continuous: DOES change

[D06] synthetic Chinchilla curve fitting (executable version of §1.3; this is a fitting unit test, not a validation of the empirical law):

alpha, beta, k_flops = 0.34, 0.28, 6            # exponents straight from §1.3's CORRECTIONS
e_true, a_true, b_true = 1.5, 400.0, 400.0      # SYNTHETIC ground truth, not the paper's fitted values
points = [(n, d, loss_surface(n, d, e_true, a_true, b_true, alpha, beta))
          for n in logspace(1e7, 1e12, 6) for d in logspace(1e8, 1e13, 6)]

e_hat, a_hat, b_hat = fit_e_a_b(points, alpha, beta)    # linear least squares (alpha,beta fixed)
assert abs(e_hat - e_true) < 1e-6 and abs(a_hat - a_true) < 1e-3 and abs(b_hat - b_true) < 1e-3
assert e_hat < min(loss for _, _, loss in points)       # irreducible loss floor
assert a_hat > 0 and b_hat > 0

c_budget = 6e21
n_star, d_star = analytic_optimum(c_budget, k_flops, alpha, beta, a_true, b_true)
n_star_100x, d_star_100x = analytic_optimum(100 * c_budget, k_flops, alpha, beta, a_true, b_true)
assert abs(n_star_100x / n_star - 100 ** (beta / (alpha + beta))) < 1e-6   # ~8.0x
assert abs(d_star_100x / d_star - 100 ** (alpha / (alpha + beta))) < 1e-6  # ~12.5x

[D04] (EOS boundary, reusing the same doc-id rule as [D02]), [D05] (padding's dual masks + no-empty-row check), and [D07] (rolling median/MAD spike detection + unconditional NaN/Inf alerts + spike/sustained-shift event classification) are folded into main() as independent assertion groups; their logic was already explained in §3.2, §3.3, and §5.4 respectively, so the code is not pasted again here.

§7 26 High-Frequency Interview Questions

Three difficulty tiers; expand each for answer key points + common pitfalls. When answering, do not repeat the main-text derivations — organize as "framework → key formulas → common mistakes".

L1 Must-Know

L2 Advanced

L3 Expert

§A Appendix: Sanity Checks

This tutorial's from-scratch implementation should satisfy the following key invariants (pure Python standard library, no third-party dependencies, seconds on CPU; script at code/pretraining_pipeline.py):

1. [D01] dedup: exact Jaccard $J(A,B)=8/12=2/3$ ($A=\{1..10\}$, $B=\{1..8,11,12\}$); the fixed-seed, 1024-permutation MinHash estimate falls within a tolerance set by sampling theory (exact equality is not asserted); LSH ($b=128,r=8$) candidate probability is far higher for the high-similarity pair ($s=2/3$) than for the low-similarity pair ($s=0.05$); union-find verifies that the transitive closure of A~B, B~C puts A and C in the same dedup cluster, which must receive the same split.
2. [D02] packing shifted labels: with tokens=[11,12,13,21,22], doc_ids=[0,0,0,1,1], independent-document semantics give labels [12,13,IGNORE,22,IGNORE]; continuous-stream semantics give [12,13,21,22,IGNORE] (the same token stream, two legitimate configurations, different labels).
3. [D03] block-diagonal mask + functional test: M[3,2]=False, M[4,0]=False, M[4,3]=True (block mode); the same coordinate in continuous-stream mode has M[3,2]=True. With $QK^\top=0$, changing doc0's values from 100 to 1000 leaves block mode's document-1 output unchanged at $0$, while continuous-stream mode's output goes from $75$ to $750$.
4. [D04] EOS boundary: on [11,12,13,EOS,21,22,EOS] / doc_ids=[0,0,0,0,1,1,1], the label content-end token (13)→EOS is kept (=EOS) and the label EOS→next document's first token (21) is masked (IGNORE) — both produced automatically by the same doc-id boundary rule.
5. [D05] padding: with valid_len=3 the loss mask is [1,1,0,0,0]; real queries cannot attend to padding keys; every attention row is non-empty (no all--inf row causing softmax NaN).
6. [D06] Chinchilla fit: from a log-spaced grid ($N\in[10^7,10^{12}]$, $D\in[10^8,10^{13}]$, 6 points each) the noiseless recovery of $E,A,B$ is within $<10^{-3}$ of the synthetic ground truth; the recovered $E$ is below all finite observed losses; the analytic $N_*,D_*$ differ from the fine-grid-search minimum by at most one grid step; at 100x compute, $N_*$ grows about $100^{0.452}\approx 8.0$x and $D_*$ about $100^{0.548}\approx 12.5$x.
7. [D07] spike monitoring: in the sequence [2.00,2.02,1.98,2.01,1.99,2.00,2.03,1.97,3.00] only the last point is flagged; a smoothly declining sequence triggers no false alarms throughout; same-magnitude single-point jumps are labeled as two different event types — spike (subsequent fall-back) vs sustained_shift (subsequent stay-high); NaN/Inf is unconditionally and immediately flagged non_finite.

Actual output of running python3 code/pretraining_pipeline.py:

[D01] dedup: exact J(A,B)=0.6667, MinHash est=0.6250 (tol 0.0884), LSH P_high=0.9939 vs P_low=0.0000; real banding candidates=[(0, 1), (1, 2)], confirmed edges=[(0, 1), (1, 2)], cluster split={0: 'test', 1: 'test', 2: 'test', 3: 'train'}  PASS
[D02] packing labels: independent-doc=[12, 13, -100, 22, -100], continuous=[12, 13, 21, 22, -100]  PASS
[D03] block mask: doc1 out unaffected by doc0 values changing (q3: 10.0->10.0, q4: 15.0->15.0); doc0 out unaffected by doc1 values changing (q0/q1/q2: 1.0/1.5/2.0); continuous mode DOES change (q3: 4.0->1502.5)  PASS
[D04] EOS boundary: labels=[12, 13, 999, -100, 22, 999, -100] (content-end->EOS kept, EOS->next-doc masked)  PASS
[D05] padding: loss_mask=[1, 1, 0, 0, 0], no attention row is empty, PAD==EOS adversarial loss_mask=[1, 1, 0, 0, 0] (real EOS target NOT masked)  PASS
[D06] Chinchilla fit: E=1.5000 A=400.00 B=400.00 (true 1.5/400.0/400.0); N*(from fitted A_hat/B_hat)=4.167e+09 matches grid (4.112e+09) and true-coefficient optimum (4.167e+09); 100x compute -> N* x8.003, D* x12.496  PASS
[D07] spike monitor: main seq flags=[8], decline seq flags=0, spike->spike vs shift->sustained_shift, NaN/Inf-> non_finite  PASS

all pretraining pipeline sanity checks passed

📚 References

• Scaling Laws (Kaplan) — Kaplan et al. (OpenAI), Scaling Laws for Neural Language Models, arXiv 2001.08361 (2020).
• Chinchilla (Hoffmann) — Hoffmann et al. (DeepMind), Training Compute-Optimal Large Language Models, arXiv 2203.15556 (2022).
• GPT-3 — Brown et al. (OpenAI), Language Models are Few-Shot Learners, arXiv 2005.14165 (2020), NeurIPS 2020.
• Gopher — Rae et al. (DeepMind), Scaling Language Models: Methods, Analysis & Insights from Training Gopher, arXiv 2112.11446 (2021).
• OPT (training logs and instability records) — Zhang et al. (Meta), OPT: Open Pre-trained Transformer Language Models, arXiv 2205.01068 (2022).
• PaLM (incl. loss-spike / checkpoint-restart practice) — Chowdhery et al. (Google), PaLM: Scaling Language Modeling with Pathways, arXiv 2204.02311 (2022).
• Mixed-precision training — Micikevicius et al. (NVIDIA/Baidu), Mixed Precision Training, arXiv 1710.03740 (2017), ICLR 2018.
• The Pile (data-pipeline design reference) — Gao et al. (EleutherAI), The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv 2101.00027 (2020).
• C4 corpus audit (extraction-quality issues) — Dodge et al., Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, arXiv 2104.08758 (2021).
• RefinedWeb (web-corpus extraction/filtering pipeline) — Penedo et al., The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, arXiv 2306.01116 (2023).
• FineWeb (large-scale web corpus + ablations) — Penedo et al. (Hugging Face), The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, arXiv 2406.17557 (2024).
• Training-data dedup (near-duplicates' downstream effects) — Lee et al. (Google), Deduplicating Training Data Makes Language Models Better, arXiv 2107.06499 (2021), ACL 2022.
• MinHash / Jaccard estimation original source — Broder, On the Resemblance and Containment of Documents, Proceedings of Compression and Complexity of Sequences (1997) (no arXiv id).
• Data mixtures / domain reweighting — Xie et al., DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, arXiv 2305.10429 (2023), NeurIPS 2023.
• Data-constrained scaling (repeated data's effect on effective D) — Muennighoff et al., Scaling Data-Constrained Language Models, arXiv 2305.16264 (2023), NeurIPS 2023.
• Llama 2 (incl. pretraining data/recipe description) — Touvron et al. (Meta), Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv 2307.09288 (2023).
• Llama 3 (incl. data-pipeline/training-recipe description) — Grattafiori et al. (Meta), The Llama 3 Herd of Models, arXiv 2407.21783 (2024).
• DeepSeek-V3 (active/total parameters and MoE training practice) — DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv 2412.19437 (2024).
• C4 / T5 (classic reference for tokenizer and corpus preprocessing) — Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, arXiv 1910.10683 (2019), JMLR 2020.