Mechanical Dreams

In this episode: • Welcome to Mechanical Dreams & The Pretraining Problem: Linda introduces the Meta FAIR paper on Self-Improving Pretraining, and Professor Norris questions why standard next-token prediction is no longer sufficient. • Breaking the Next-Token Paradigm: Linda explains the shift from next-token prediction to prefix-conditioned suffix generation, arguing that post-training safety ali...

In this episode: • Introduction: What is a Duplicate?: Professor Norris and Linda introduce the paper Scale Dependent Data Duplication and discuss the core question of what really counts as a duplicate for a language model. • The Emergence of Semantics: Linda breaks down how larger, more capable models begin to treat semantic equivalents like translations as exact duplicates, and Norris reacts to...

In this episode: • Welcome and Introduction: Professor Norris and Linda introduce the podcast and the topic of the week, discussing the general concept of representation degeneration in neural language models. • The Culprit: Rare Tokens: Linda explains the paper's core empirical finding: rare token embeddings degenerate first and drag the rest of the tokens into a narrow cone. • Adaptive Gradient...

In this episode: • Introduction: The Gold Standard in Question: Professor Norris and Linda introduce the episode's paper from DeepMind, setting the stage by defining perplexity and explaining why it is universally used to evaluate language models. • The Copy Task and the Illusion of Confidence: Linda breaks down the theoretical proof using a bitstring copy task, explaining how high confidence in a...

In this episode: • Introduction to Downstream Scaling Laws: Linda and Professor Norris introduce the paper and discuss the limitations of traditional parametric scaling laws for predicting downstream task performance. • The Token-Level Secret: Linda explains how NeuNeu uses token-level probabilities instead of average validation loss to capture critical distributional signals. • Architecture Deep...

In this episode: • Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models. • Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers. •...

In this episode: • Welcome & Introduction: Professor Norris and Linda welcome the listeners. Linda introduces the paper of the week, teasing the unexpected comeback of non-linear RNNs. • The Expressivity Gap: Linear vs. Non-Linear RNNs: The hosts discuss how linear RNNs like Mamba and Gated DeltaNet dominate due to their efficiency, but fundamentally lack the expressive power for complex state-tra...

In this episode: • Chapter 1: Introduction to the Bottleneck: Linda introduces the paper and the general concept of the LM head. Professor Norris expresses initial skepticism about revisiting the softmax bottleneck. • Chapter 2: Expressivity vs. Optimization: The hosts discuss how the paper shifts the focus from the classical expressivity limitation to a fundamental optimization problem. • Chapter...

In this episode: • The Context Window Illusion: Norris and Linda introduce the episode and the paper, discussing why million-token context windows don't automatically solve reasoning tasks. • The Math of Score Dilution: Linda dives into the theoretical bottleneck of static self-attention, explaining why the target-distractor margin must scale logarithmically. • Query-Only Test-Time Training: Linda...

In this episode: • Introduction to State-Tracking: Linda and Professor Norris introduce the paper and discuss the historical context of state-tracking in sequence models. • The Next-Token Prediction Testbed: The hosts discuss how the authors used Python REPL traces with print statements to evaluate models using next-token prediction instead of sequence-to-sequence. • DeltaNet Triumphs Over Transfo...

In this episode: • Dessert Before Vegetables?: Professor Norris and Linda introduce the concept of Curriculum Learning in LLMs and discuss why the intuitive idea of saving the best data for last has historically failed to produce significant results. • The Invisible antagonist: Learning Rate Decay: Linda reveals the paper's core insight: standard learning rate schedules decay to near-zero just as...

In this episode: • Welcome & The End of Vibe Coding?: Linda introduces GLM-5 and the paradigm shift from passive vibe coding to autonomous agentic engineering. • Architecture & DeepSeek Sparse Attention: Professor Norris and Linda examine the 744B parameter model and how transitioning from dense to sparse attention drastically cuts compute costs. • Asynchronous RL and the Slime Framework: A deep d...

In this episode: • Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism. • The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function. • The One-Line Fix and S...

In this episode: • Welcome and Introduction: Professor Norris and Linda introduce the episode and the paper of the week: 'Backward Gradient Normalization in Deep Neural Networks'. • The Ghost of Gradients Past: A discussion on the classic vanishing and exploding gradient problems, and why existing solutions like Batch Normalization and ResNets still leave room for improvement. • Unpacking Backward...

In this episode: • The PreNorm Dilution Problem: Professor Norris and Linda introduce the episode and discuss the fundamental limitations of standard residual connections, focusing on the unbounded magnitude growth caused by PreNorm. • Attention Residuals and the Time-Depth Duality: Linda introduces the core concept of Full Attention Residuals, treating network depth like sequence length. Professo...

In this episode: • Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper. • The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information. • Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where. • Empirical Triumphs: Reviewing t...

In this episode: • Introduction to Precision in Scaling Laws: Linda introduces the new paper which adds precision as a third variable to the Chinchilla scaling laws. Professor Norris reflects on how precision is usually treated as an afterthought. • The Post-Training Quantization Paradox: The hosts discuss the surprising finding that overtraining models on too much data actually makes them degrade...

In this episode: • Introduction: The Transformer's Kryptonite: Professor Norris jokes about Transformers solving everything, but Linda introduces a new paper that challenges their ability to perform basic state tracking efficiently. They set the stage by distinguishing between the well-known Out-of-Distribution failures and the paper's focus on In-Distribution data efficiency. • The Setup: Modulo...

In this episode: • A Noble Introduction: Professor Norris makes a pun about aristocracy while Linda introduces the paper 'NOBLE' from Canva Research, setting the stage for a discussion on accelerating Transformer pretraining. • The Linear Collapse Problem: Linda explains why standard LoRA doesn't work for pretraining from scratch, and Norris helps clarify the difference between parameter-efficient...

In this episode: • Welcome to the Hardware Lottery: Professor Norris and Linda introduce the episode's focus: FlashAttention-4. They set the stage by discussing the arrival of NVIDIA's Blackwell architecture and why existing optimization techniques suddenly hit a wall. • The Asymmetry Problem: Linda explains the concept of 'Asymmetric Hardware Scaling' found in the B200 GPUs, where tensor cores do...

In this episode: • The Multi-Million Dollar NaN: Linda introduces the paper 'An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence' by Zhang et al., setting the stage with the high stakes of expensive pretraining runs failing. Professor Norris expresses skepticism that simple 'bad data' is the root cause of complex divergences. • The Toxic Five Tokens: The hosts discuss the paper's...

In this episode: • Introduction: Do We Really Need Another Phase?: Professor Norris jokingly laments the ever-expanding terminology of LLM training, while Linda introduces the paper on 'Midtraining' as a distinct, intermediate phase between pretraining and post-training. • The Mechanism: Building a Distributional Bridge: Linda explains the core theory: midtraining isn't just 'cooling down,' but sh...

In this episode: • Introduction: The Never-Ending Normalization Wars: Professor Norris and Linda kick off the episode. Norris cracks a joke about how normalization layers are like seasoning—too little and it's bland, too much and you ruin the dish. Linda introduces the paper 'SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm', setting the stage for a discussion on the fundamental trad...

In this episode: • Welcome to the ÜberWeb: Professor Norris and Linda introduce the episode's focus: the 'ÜberWeb' paper by DatologyAI, setting the stage for a discussion on the challenges of training high-quality multilingual models on a massive scale. • The Curse That Wasn't: The hosts debate the 'curse of multilinguality,' with Linda explaining the paper's central thesis: that performance degra...

In this episode: • Introduction: The Infinite Loop: Professor Norris and Linda introduce the episode's topic: the phenomenon of reasoning models getting stuck in repetitive loops. Norris jokes about his own lectures looping, while Linda introduces the paper 'Wait, Wait, Wait... Why Do Reasoning Models Loop?' and the context of Chain-of-Thought reasoning. • The Distillation Mystery: Linda presents...