Mastering Language Models: From Architecture to Optimization

In the finale of Topic 7 and the series, Maya and Leo unpack a 2025 systematic review of optimization methods for machine learning — the optimizer that decides whether a ninety-day run converges. They map the gradient-based family as a lineage of flaw-fixes (SGD, momentum, Nesterov, AdaGrad, RMSProp, Adam, AdamW), touch second-order/quasi-Newton and large-batch methods (LARS, LAMB), and perform th...

Maya and Leo trace AutoML from grid and random search through Bayesian optimization, neural architecture search, and automated pipeline construction (TPOT, AutoGluon, Auto-Keras), then ask where automated search beats expert hand-tuning when both draw from the same fixed ninety-day compute budget. The episode's spine: the unit of automation keeps growing, the real enemy is the cost of one evaluati...

The Pile reframes training data as a design choice. EleutherAI assembled an 825-gibibyte English corpus from twenty-two diverse sub-datasets — books, code, biomedical and physics papers, legal opinions, dialogue, and a filtered slice of the open web — and showed that diversity improves a model's general, cross-domain ability. In a controlled comparison against raw and cleaned web crawl, the divers...

Maya and Leo follow the sparsely-gated Mixture-of-Experts idea out of language and into scientific machine learning. The paper, MoE-POT, is an operator transformer pre-trained across many families of partial differential equations — fluid flow, heat, waves — where a layer-wise router sends each chunk of a physical field to a few specialized experts plus a couple of always-on shared experts. They e...

Maya and Leo unpack the 2017 paper that started sparse scaling. They walk from the wall it was hitting — capacity and compute welded together in dense models — to the central move: a Mixture-of-Experts layer where a noisy top-k gating network routes each token to just a few of up to thousands of experts, so total capacity can grow a thousand-fold without compute-per-word following. They explain ex...

Maya and Leo open Topic 7 by going underneath massive models to the machinery that makes them possible. Using one small lab training a large sparse model on a fixed ninety-day compute budget, they explain sparsity and Mixture-of-Experts, the router and load balancing, why diverse training data is a design choice, how AutoML automates tuning, and what it takes for any of it to converge — staging th...

Maya and Leo unpack DeepSeek-V4-Flash as an efficiency stack for million-token context: hybrid compressed attention, sparse expert activation, low-precision serving, and specialist distillation. They stage the field's real arguments — how hard to compress long-context memory, and when a local Flash-class model should escalate to a Pro-class or hosted one — and land on a managed-memory mental model...

Maya and Leo unpack DeepSeek-V3 as a sparse, infrastructure-heavy route to frontier open-weight capability. They explain MoE routing, MLA cache compression, auxiliary-loss-free load balancing, DualPipe, FP8 training, multi-token prediction, GRPO post-training, benchmark caveats, and why deployment complexity matters for teams choosing between local, hosted, dense, and sparse models. Sources: • Dee...

Maya and Leo unpack Meta's Llama 3 Herd of Models paper as a deployment portfolio: dense architecture, massive curated data, post-training with SFT, rejection sampling and DPO, long-context and tool-use training, safety rails, and a staged debate over the trade-offs of open-weight release. Sources: • The Llama 3 Herd of Models: https://arxiv.org/pdf/2407.21783 • Llama 2: Open Foundation and Fine-T...

This episode unpacks the Llama 2 paper as more than a model announcement: it is a stack of base weights, chat tuning, RLHF, safety work, evaluation caveats, and release terms. Maya and Leo connect the paper to a practical on-device assistant team deciding how to use open weights responsibly. Sources: • Llama 2: Open Foundation and Fine-Tuned Chat Models: https://arxiv.org/pdf/2307.09288 • Llama 2...

Maya and Leo introduce Topic 6 by mapping open-weight and fine-tuned model families as deployable engineering components. Using an on-device coding and data-analysis assistant, they explain Llama, fine-tuning, deployment envelopes, private evaluation, ecosystem trade-offs, and why sparse models like DeepSeek complicate the open-model frontier. Sources: • Llama 2: Open Foundation and Fine-Tuned Cha...

This episode closes Topic 5 with John Schulman's Berkeley EECS colloquium on RLHF progress and challenges — the architect of PPO and ChatGPT-era preference tuning grading his own pipeline. Maya and Leo walk the progress column (comparisons make negative feedback usable where intuition outruns specification) and four challenge landmarks: the Applause Meter, the Tired Jury, the Smooth Talker, and th...

Maya and Leo close in on the repair episode of the RLHF arc: VRPO, a variance-reduced preference optimization method for fine-tuning language models when human labels are scarce and the Bradley-Terry assumptions are misspecified. Through a water-utility calibration story, they unpack the control-variate maneuver — keep the human-labeled loss in charge, subtract an auxiliary judge's prediction on e...

Maya and Leo take a deep breath after the method war and inspect the instrument every RLHF pipeline depends on: the reward model. Through the lens of RLHF Deciphered, they map the gap between the oracular reward nobody has and the fitted surface everyone trains — the coverage holes in human feedback, the misgeneralized scores an optimizer happily paves into behavior, the whole-answer labels that s...

Maya and Leo unpack Direct Preference Optimization, the 2023 paper whose napkin-worthy algebra showed the reward model was hiding inside the language model all along. They walk the old two-stage RLHF pipeline, then the substitution that cancels the reward variable and leaves a supervised-looking classification loss, the implicit reward you can read off the tuned model's margin over its reference,...

Maya and Leo dig into Constitutional AI, the Anthropic paper that swaps many human harmlessness labels for a short written constitution: the model critiques and rewrites its own risky answers, then an AI judge compares candidate replies against the principles to drive reinforcement learning from AI feedback. Using a healthcare scheduling assistant, they show why critique-before-revision matters, w...

Maya and Leo dig into Anthropic's helpful-and-harmless RLHF paper: two opposite data-collection payrolls, one preference model serving two masters, the weekly online refresh that keeps the judge informed, the split-judge robustness test that exposes reward gaming, and a staged fight over whether the alignment tax is real. Sources: • Training a Helpful and Harmless Assistant with Reinforcement Lear...

Maya and Leo dig into the InstructGPT paper — the moment the human-feedback recipe grew from a summarization trick into the way assistants get made. They walk the pipeline as three stations and a punch list (the Apprenticeship, the Ranking Desk, the Governor, the Punch List), stage the scale-versus-feedback argument over the famous result that humans preferred a 1.3B-parameter aligned model to the...

Maya and Leo dig into OpenAI's Learning to Summarize from Human Feedback — the paper where pairwise human picks replaced reference matching as the training target. They walk the pipeline as three stations (the Two-Card Choice, the Borrowed Judge, the Tether), stage the real fight between preference optimization and cheap reproducible metrics, and end on the over-optimization curve where the judge'...

Maya and Leo open the Topic 5 deep dives with the paper that made preference optimization practical: Proximal Policy Optimization. Starting from a physical-therapy brace that stops paying out range past a set angle, they unpack why step size is existential when a policy generates its own training data, how the clipped probability ratio and the pessimistic minimum make updates safe to repeat, why b...

A topic overview of RLHF: how human comparisons become preference data, how reward models and cautious optimization steer assistant behavior, why the PPO pipeline and DPO represent a genuine method war, and where feedback loops can be gamed or go brittle. Sources: • Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347 • Learning to Summarize with Human Feedback: https://arxiv....

Topic 4 closes by stretching specialization across time. Maya opens with an interpreter in Lisbon losing words in her own first language — first-language attrition as the human face of catastrophic forgetting — and the survey behind the episode maps the machine version: erosion with good manners, where the model stays fluent while old domains, formats, and safety behaviors quietly slip. The hosts...

The paper that asks how few bits an adapter can survive on. Maya opens with a mosaic master firing a portrait in four tile shades, and the analogy turns out to be the method: LowRA pushes LoRA fine-tuning below two bits per parameter through three deliberate decisions — the Palette (which values the codes stand for), the Cut Lines (where bucket boundaries sit), and the Bit Budget (where bits get s...

The paper that put large-model fine-tuning on a single GPU. Maya and Leo open QLoRA's central rule — read the compressed thing, write somewhere else — and follow gradients through a frozen four-bit base into full-precision LoRA adapters. Along the way: NF4's bell-curve-shaped buckets, double quantization's compress-the-labels trick, paged optimizers as the relief valve that saves hour-nine runs, a...

The paper that made fine-tuning feel modular. Maya and Leo open up LoRA's central trick — freeze the pretrained weights and learn the update as the product of two thin matrices, around sixty-five thousand trainable numbers standing in for sixteen million — then follow it through the merge fork (flatten for zero-overhead serving, or keep adapters swappable on one frozen base), the rank and alpha di...