Mechanical Dreams

In this episode: • Introduction: Hitting the Data Wall: Professor Norris and Linda introduce the episode's paper, 'OPUS', and discuss the looming 'Data Wall' where high-quality public text is exhausted, necessitating a shift from more tokens to better tokens. • The Flaw in Current Data Selection: The hosts debate existing methods, contrasting static filters like FineWeb-Edu with dynamic selection....

In this episode: • Introduction: The Optimizer Zoo: Professor Norris and Linda introduce the topic of optimization in LLMs, joking about the explosion of new optimizers before introducing the paper of the week: TEON. • The Muon Foundation: Linda recaps the Muon optimizer, explaining how it uses orthogonalization to prevent gradient rank collapse, while Norris questions its limitations regarding la...

In this episode: • Introduction: The Weight Decay Dilemma: Professor Norris and Linda introduce the episode's topic: Cautious Weight Decay. They discuss the historical context of weight decay as a regularization technique and why standard approaches might be accidentally sabotaging model learning. • The Mechanism: To Decay or Not to Decay?: Linda explains the core algorithm of Cautious Weight Deca...

In this episode: • Introduction: The Alchemy of Training: Professor Norris laments the 'black magic' of hyperparameter tuning, and Linda introduces the paper 'Predictable Scale: Part I, Step Law' which promises to turn that alchemy into science. • The Million-Hour Experiment: The hosts discuss the unprecedented scale of the study, involving 3,700 models and nearly one million H800 GPU hours, to ma...

In this episode: • Introduction: The Heavy Cost of Curvature: Professor Norris and Linda introduce the paper 'A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs' by researchers at Meta and UMD, setting the stage by discussing why measuring the Hessian matrix is a computational nightmare for large models. • The Proposal: Critical Sharpness: Linda explains the...

In this episode: • Introduction: The Disconnect: Professor Norris and Linda introduce the paper 'Challenges and Research Directions for Large Language Model Inference Hardware' by Ma and Patterson, discussing the widening gap between academic architecture research and industry reality. • The Inference Crisis: Prefill vs. Decode: The hosts break down why LLM inference is fundamentally different fro...

In this episode: • To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate. • The Inductive Bias Paradox: Linda explains t...

In this episode: • Introduction: The Mystery of the Straight Line: Professor Norris and Linda introduce the paper 'The Quantization Model of Neural Scaling' by Michaud et al., setting the stage by discussing the ubiquity of power laws in deep learning and the puzzle of why scaling curves are so predictable. • The Quantization Hypothesis: Linda explains the core theory that neural network knowledge...

In this episode: • Introduction: The Wait for Tokens: Professor Norris and Linda introduce the episode's paper, EAGLE-3, and discuss the persistent bottleneck of autoregressive generation costs in modern LLMs. • The Speculative Ceiling: Linda explains how previous speculative sampling methods like EAGLE hit a performance wall where adding more training data failed to improve the draft model, ident...

In this episode: • The Memory Bottleneck: Professor Norris and Linda introduce the paper 'Conditional Memory via Scalable Lookup' and debate the inefficiency of using expensive neural computation to simulate simple knowledge retrieval. • Engram: N-grams Strike Back: Linda breaks down the 'Engram' module, explaining how it uses hashed N-grams and context-aware gating to inject static embeddings dir...

In this episode: • Introduction: Is Shannon Information Theory Broken?: Professor Norris and Linda introduce the episode, with Norris expressing skepticism about challenging the foundations of information theory. Linda introduces the paper 'From Entropy to Epiplexity' and the premise that traditional theory fails to account for computational bounds. • The Paradox of Deterministic Creation: The hos...

In this episode: • Introduction: The Alchemy of Training: Professor Norris and Linda introduce the episode, joking about the 'black art' of hyperparameter tuning before unveiling the paper of the week: 'Completed Hyperparameter Transfer' by researchers at Apple. • Beyond Width: The Limits of muP: Linda explains the background of the Maximal Update Parametrization (muP) and why scaling only across...

In this episode: • Introduction: The Optimizer Menagerie: Professor Norris and Linda kick off the episode by discussing the explosion of new optimizers in the LLM space. Linda introduces 'NorMuon,' a paper from Georgia Tech and Microsoft that attempts to bridge the gap between the industry standard, AdamW, and the geometric newcomer, Muon. • The Geometry Problem: Why Adam and Muon Fall Short: Lind...

In this episode: • The GPU Bill Blues: Professor Norris laments the exorbitant cost of training large models, setting the stage for Linda to introduce the episode's focus: 'Dion: Distributed Orthonormalized Updates' by researchers from Microsoft and Harvard. • Muon's Heavy Lifting: Linda explains the predecessor, the Muon optimizer, and its orthonormalization benefits. Norris questions why a new m...

In this episode: • Is Query Redundant?: Linda introduces a provocative paper suggesting a core part of the Transformer attention mechanism, the Query matrix, might be unnecessary. Professor Norris expresses his trademark skepticism about simplifying such a fundamental component. • The Usual Suspects: Q, K, and V: Linda provides a quick, intuitive refresher on the roles of Query, Key, and Value mat...

In this episode: • Why Does Seed 42 Work Best?: Linda introduces a paper that tries to answer a classic machine learning question: why does the random seed have such a big impact on training? Professor Norris laments that this is a problem as old as neural networks themselves. • A Roadmap for Training: Linda explains the paper's novel approach of using a Hidden Markov Model to turn messy training...

In this episode: • Why a Good Pre-trainer Isn't Always a Good Finetuner: The hosts introduce the puzzle of pre-training: why doesn't a lower cross-entropy loss always guarantee better performance after fine-tuning? They set the stage for today's paper which proposes a new perspective. • Are We Covering Our Bases? The Coverage Principle: Linda explains the paper's central concept of 'coverage,' a m...

In this episode: • The Art and Science of Scaling RL: Professor Norris and Linda introduce today's topic, a new paper from Meta that aims to make training large models with reinforcement learning more predictable and scientific. • More Art than Science: Linda explains why scaling Reinforcement Learning is so difficult compared to pre-training, highlighting the lack of predictive scaling laws and t...

In this episode: • The Frozen Brains of AI: Linda introduces the problem of static LLMs and the challenge of 'catastrophic forgetting.' Professor Norris provides historical context on this long-standing issue in AI and introduces the day's paper on continual learning. • Why Can't Models Just Keep Learning?: The hosts discuss traditional approaches to continual learning, like data replay and regula...