Mechanical Dreams

In this episode: • Introduction: The Transformer's Kryptonite: Professor Norris jokes about Transformers solving everything, but Linda introduces a new paper that challenges their ability to perform basic state tracking efficiently. They set the stage by distinguishing between the well-known Out-of-Distribution failures and the paper's focus on In-Distribution data efficiency. • The Setup: Modulo...

In this episode: • A Noble Introduction: Professor Norris makes a pun about aristocracy while Linda introduces the paper 'NOBLE' from Canva Research, setting the stage for a discussion on accelerating Transformer pretraining. • The Linear Collapse Problem: Linda explains why standard LoRA doesn't work for pretraining from scratch, and Norris helps clarify the difference between parameter-efficient...

In this episode: • Welcome to the Hardware Lottery: Professor Norris and Linda introduce the episode's focus: FlashAttention-4. They set the stage by discussing the arrival of NVIDIA's Blackwell architecture and why existing optimization techniques suddenly hit a wall. • The Asymmetry Problem: Linda explains the concept of 'Asymmetric Hardware Scaling' found in the B200 GPUs, where tensor cores do...

In this episode: • The Multi-Million Dollar NaN: Linda introduces the paper 'An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence' by Zhang et al., setting the stage with the high stakes of expensive pretraining runs failing. Professor Norris expresses skepticism that simple 'bad data' is the root cause of complex divergences. • The Toxic Five Tokens: The hosts discuss the paper's...

In this episode: • Introduction: Do We Really Need Another Phase?: Professor Norris jokingly laments the ever-expanding terminology of LLM training, while Linda introduces the paper on 'Midtraining' as a distinct, intermediate phase between pretraining and post-training. • The Mechanism: Building a Distributional Bridge: Linda explains the core theory: midtraining isn't just 'cooling down,' but sh...

In this episode: • Introduction: The Never-Ending Normalization Wars: Professor Norris and Linda kick off the episode. Norris cracks a joke about how normalization layers are like seasoning—too little and it's bland, too much and you ruin the dish. Linda introduces the paper 'SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm', setting the stage for a discussion on the fundamental trad...

In this episode: • Welcome to the ÜberWeb: Professor Norris and Linda introduce the episode's focus: the 'ÜberWeb' paper by DatologyAI, setting the stage for a discussion on the challenges of training high-quality multilingual models on a massive scale. • The Curse That Wasn't: The hosts debate the 'curse of multilinguality,' with Linda explaining the paper's central thesis: that performance degra...

In this episode: • Introduction: The Infinite Loop: Professor Norris and Linda introduce the episode's topic: the phenomenon of reasoning models getting stuck in repetitive loops. Norris jokes about his own lectures looping, while Linda introduces the paper 'Wait, Wait, Wait... Why Do Reasoning Models Loop?' and the context of Chain-of-Thought reasoning. • The Distillation Mystery: Linda presents...

In this episode: • Introduction: Hitting the Data Wall: Professor Norris and Linda introduce the episode's paper, 'OPUS', and discuss the looming 'Data Wall' where high-quality public text is exhausted, necessitating a shift from more tokens to better tokens. • The Flaw in Current Data Selection: The hosts debate existing methods, contrasting static filters like FineWeb-Edu with dynamic selection....

In this episode: • Introduction: The Optimizer Zoo: Professor Norris and Linda introduce the topic of optimization in LLMs, joking about the explosion of new optimizers before introducing the paper of the week: TEON. • The Muon Foundation: Linda recaps the Muon optimizer, explaining how it uses orthogonalization to prevent gradient rank collapse, while Norris questions its limitations regarding la...

In this episode: • Introduction: The Weight Decay Dilemma: Professor Norris and Linda introduce the episode's topic: Cautious Weight Decay. They discuss the historical context of weight decay as a regularization technique and why standard approaches might be accidentally sabotaging model learning. • The Mechanism: To Decay or Not to Decay?: Linda explains the core algorithm of Cautious Weight Deca...

In this episode: • Introduction: The Alchemy of Training: Professor Norris laments the 'black magic' of hyperparameter tuning, and Linda introduces the paper 'Predictable Scale: Part I, Step Law' which promises to turn that alchemy into science. • The Million-Hour Experiment: The hosts discuss the unprecedented scale of the study, involving 3,700 models and nearly one million H800 GPU hours, to ma...

In this episode: • Introduction: The Heavy Cost of Curvature: Professor Norris and Linda introduce the paper 'A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs' by researchers at Meta and UMD, setting the stage by discussing why measuring the Hessian matrix is a computational nightmare for large models. • The Proposal: Critical Sharpness: Linda explains the...

In this episode: • Introduction: The Disconnect: Professor Norris and Linda introduce the paper 'Challenges and Research Directions for Large Language Model Inference Hardware' by Ma and Patterson, discussing the widening gap between academic architecture research and industry reality. • The Inference Crisis: Prefill vs. Decode: The hosts break down why LLM inference is fundamentally different fro...

In this episode: • To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate. • The Inductive Bias Paradox: Linda explains t...

In this episode: • Introduction: The Mystery of the Straight Line: Professor Norris and Linda introduce the paper 'The Quantization Model of Neural Scaling' by Michaud et al., setting the stage by discussing the ubiquity of power laws in deep learning and the puzzle of why scaling curves are so predictable. • The Quantization Hypothesis: Linda explains the core theory that neural network knowledge...

In this episode: • Introduction: The Wait for Tokens: Professor Norris and Linda introduce the episode's paper, EAGLE-3, and discuss the persistent bottleneck of autoregressive generation costs in modern LLMs. • The Speculative Ceiling: Linda explains how previous speculative sampling methods like EAGLE hit a performance wall where adding more training data failed to improve the draft model, ident...

In this episode: • The Memory Bottleneck: Professor Norris and Linda introduce the paper 'Conditional Memory via Scalable Lookup' and debate the inefficiency of using expensive neural computation to simulate simple knowledge retrieval. • Engram: N-grams Strike Back: Linda breaks down the 'Engram' module, explaining how it uses hashed N-grams and context-aware gating to inject static embeddings dir...

In this episode: • Introduction: Is Shannon Information Theory Broken?: Professor Norris and Linda introduce the episode, with Norris expressing skepticism about challenging the foundations of information theory. Linda introduces the paper 'From Entropy to Epiplexity' and the premise that traditional theory fails to account for computational bounds. • The Paradox of Deterministic Creation: The hos...

In this episode: • Introduction: The Alchemy of Training: Professor Norris and Linda introduce the episode, joking about the 'black art' of hyperparameter tuning before unveiling the paper of the week: 'Completed Hyperparameter Transfer' by researchers at Apple. • Beyond Width: The Limits of muP: Linda explains the background of the Maximal Update Parametrization (muP) and why scaling only across...

In this episode: • Introduction: The Optimizer Menagerie: Professor Norris and Linda kick off the episode by discussing the explosion of new optimizers in the LLM space. Linda introduces 'NorMuon,' a paper from Georgia Tech and Microsoft that attempts to bridge the gap between the industry standard, AdamW, and the geometric newcomer, Muon. • The Geometry Problem: Why Adam and Muon Fall Short: Lind...

In this episode: • The GPU Bill Blues: Professor Norris laments the exorbitant cost of training large models, setting the stage for Linda to introduce the episode's focus: 'Dion: Distributed Orthonormalized Updates' by researchers from Microsoft and Harvard. • Muon's Heavy Lifting: Linda explains the predecessor, the Muon optimizer, and its orthonormalization benefits. Norris questions why a new m...