Mastering Language Models: From Architecture to Optimization

Topic 4 opens with the move that defines modern model specialization: freeze the giant base model and train a tiny low-rank update beside it. Maya and Leo map the topic's landmarks — the Rank Knob, the Precision Floor, the What-Got-Worse Test, and the Long Haul — and stage the field's real arguments: PEFT-everywhere versus full fine-tuning's capacity case, and how many bits the frozen base can los...

The final episode of Topic 3 closes the loop by picking a fight with Topic 2. Chinchilla's compute-optimal recipe balances model size against training tokens under a training budget — but deployed reasoning systems don't pay for one forward pass per task. They sample candidates and vote, run verifiers, search, retry. Today's paper, Test-Time Scaling Makes Overtraining Compute-Optimal, writes down...

Episode eight of Topic 3 steps out of the machine room. After seven episodes of chips, memory, cables, and schedules, the bottleneck moves to the post-training bill — and the paper attacking it is almost suspiciously simple. SSD — Embarrassingly Simple Self-Distillation — samples code solutions from a model under chosen temperature and truncation settings, then fine-tunes the same model on those r...

Episode seven of Topic 3 stays inside the GPU for the sequel. FlashAttention ended attention's memory commute, and the same author's audit found the chip still far from busy — so FlashAttention-2 re-divides the labor instead of the math: less slow non-matmul bookkeeping, a single attention head's work split across many thread blocks so long-sequence runs fill the machine, and warps exchanging less...

Episode six of Topic 3 stops spreading work across machines and crawls inside a single GPU. FlashAttention's accusation is that attention was slow for the wrong reason: not too much math, too much traffic — the quadratic score matrix hauled back and forth between high-bandwidth memory and the tiny on-chip SRAM beside the compute units. Maya and Leo open at a laundromat where the machines were neve...

Episode five of Topic 3 steps back from single techniques to the whole system. Maya and Leo open at a container port at dawn — the cranes are the postcard, but the slowest gate decides when the ship leaves — and use a 2025 ACM survey to define a training architecture as a distributed system with machine-learning math inside it. They walk six harbor-named stops where real runs get caught: the Chann...

Episode four of Topic 3 is the sequel to ZeRO's argument: what happens when sharding wins and moves into PyTorch as Fully Sharded Data Parallel. Maya and Leo open in a machine shop where no parts live at the bench — crates arrive exactly when a job needs them — and follow FSDP's loop of gathering full parameters for one wrapped block, computing locally, and letting the copy go. Then the four negot...

Episode three of Topic 3 stops cutting the model. After two episodes of pipeline and tensor surgery, ZeRO — the Zero Redundancy Optimizer — asks the almost-rude question: what if the real waste was never the math, but the copies? Maya and Leo open in a town where eight libraries stock identical collections and ledgers fatter than the books, then follow the sharding ladder — optimizer states, gradi...

Episode two of Topic 3 goes inside the layer. When a single Transformer layer is too big for one chip, no pipeline schedule can save you — so Megatron-LM cuts the matrix multiplications themselves across GPUs, column-wise then row-wise, with an all-reduce 'huddle' only where partial results must meet. Maya and Leo walk the feed-forward and attention splits with their sixty-four-GPU team, then swap...

The first deep dive of Topic 3 takes on the bluntest bottleneck: the model does not fit on one device. Maya and Leo unpack GPipe's move — slice the layer stack into stages, stream microbatches through them like trays down a sandwich line, and re-materialize activations instead of storing them — then stage the field's real argument between pipeline and tensor parallelism: idle bubbles versus consta...

Topic 3 opens with a map of the bottlenecks that decide whether a hundred-billion-parameter model can be trained at all: model-state memory, activation memory, GPU-to-GPU communication, pipeline bubbles, and the data movement inside a single chip. Maya and Leo stage the field's real argument — partition the model versus shard the redundant states — introduce the four levers (copy, slice, split, sh...

Deep dive into Muennighoff et al.'s Scaling Data-Constrained Language Models (2023) — the paper that asks what happens when the balanced scaling recipe demands more fresh, high-quality text than exists. Maya and Leo walk the usable shelf (why the responsibly trainable internet is far smaller than the internet), the second pass (epochs and repetition), and the repetition discount (a few passes are...

Deep dive into Hoffmann et al.'s Training Compute-Optimal Large Language Models (2022) — the Chinchilla paper that re-measured the parameters-versus-tokens trade-off and found a generation of large models undertrained. Maya and Leo walk the three landmarks: the rebalance — under a fixed compute budget, model size and training tokens should scale roughly together; the rematch — a seventy-billion-pa...

Deep dive into Kaplan et al.'s Scaling Laws for Neural Language Models (2020), the paper that made giant training runs forecastable. Maya and Leo walk the three landmarks: the ruler — loss falls along smooth power laws in parameters, data, and compute, so cheap pilot runs predict frontier runs; the early exit — larger models learn more per token, so a fixed budget should buy a huge model trained o...

Topic 2 opens with the question the Transformer made urgent: once you can build big, how should one fixed training budget be split between model size, training tokens, and data quality? Maya and Leo stage the scale-first versus compute-optimal argument in its strongest forms, introduce the smooth-curve predictability of scaling laws, the four interacting knobs of scale, and the two-bills view of t...

Topic 1 closes with the 2017 paper's confession answered. Kimi Linear, from Moonshot AI's Kimi Team, claims a first: a mostly-linear attention architecture that beats full attention under matched training runs. Maya builds the machine in four landmarks — the Board (a fixed-size memory that never grows), the Eraser (the delta rule's overwrite-don't-pile update), the Knobs (a learned forgetting dial...

The first deep dive of the series opens the 2017 Transformer paper itself. Maya walks the machine in three stations — the Matchmaker (query-key-value lookup), the Committee (eight specialist attention heads), and the Chord (wave-stamped word order) — while Leo brings the receipts: a two-point BLEU jump over every published system, base-model training in about twelve hours on eight GPUs, and an abl...

Topic 1 opens the series at the foundation: how 'Attention Is All You Need' replaced step-by-step recurrence with self-attention — every token seeing every other token in one parallel hop — and why that single move made large-scale pre-training possible. Maya and Leo build the topic's shared mental models (attention as content-based lookup, the two bills of training versus serving, architectures a...

Maya and Leo open the series with the map: seven stops from the Transformer blueprint to the machinery under massive models, anchored by a three-person startup building an insurance-claims assistant on eight GPUs. They lay out the mental models every LLM expert shares — trust curves, find the bottleneck, separate capability from behavior — then stage the field's cleanest fight on air: bigger model...