Mechanical Dreams

In this episode: • Welcome and the Over-smoothing Problem: Professor Norris and Linda introduce the episode and discuss how deep Transformers suffer from over-smoothing, losing initial token-level information in later layers. • Introducing ResFormer and Value Residuals: Linda explains the core mechanism of ResFormer, which adds a residual connection specifically to the Value vectors from the first...

In this episode: • Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting. • Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure featu...

In this episode: • Introduction to the Compute Divide: Linda and Professor Norris introduce the podcast and discuss the massive computational barriers in modern LLM pretraining before introducing the HRM-Text paper. • Biological Inspiration and the HRM Architecture: The hosts discuss how the human brain's frontoparietal loop inspired the dual-timescale Hierarchical Recurrent Model, breaking down t...

In this episode: • Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice. • Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to toler...

In this episode: • Introduction to the Edge of Stability: Professor Norris and Linda introduce the paper and the surprising behavior of full-batch gradient descent. • Progressive Sharpening: Linda explains how gradient descent naturally navigates towards steeper areas of the loss landscape, increasing sharpness. • Surviving the Edge: The hosts discuss how neural networks avoid catastrophic diverge...

In this episode: • Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models. • The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activati...

In this episode: • Welcome & The Quest for Lifetime Memory: Linda introduces the paper on Memory Sparse Attention (MSA) and sets the stage by comparing current LLM context windows to human lifelong memory capacity. • The Context Length Bottleneck: Professor Norris and Linda discuss why current approaches like full attention, fixed-size memory states (RNNs), and traditional RAG systems struggle to...

In this episode: • Welcome and the Post-hoc Problem: Linda introduces the paper and the hosts discuss why post-hoc unlearning methods fall short against adversarial attacks. • Token vs. Document Filtering: An exploration of why token-level filtering acts as a scalpel compared to the blunt instrument of document filtering. • Scaling Labels with SAEs: The hosts discuss how the authors use Sparse Aut...

In this episode: • Welcome to Mechanical Dreams & The Pretraining Problem: Linda introduces the Meta FAIR paper on Self-Improving Pretraining, and Professor Norris questions why standard next-token prediction is no longer sufficient. • Breaking the Next-Token Paradigm: Linda explains the shift from next-token prediction to prefix-conditioned suffix generation, arguing that post-training safety ali...

In this episode: • Introduction: What is a Duplicate?: Professor Norris and Linda introduce the paper Scale Dependent Data Duplication and discuss the core question of what really counts as a duplicate for a language model. • The Emergence of Semantics: Linda breaks down how larger, more capable models begin to treat semantic equivalents like translations as exact duplicates, and Norris reacts to...

In this episode: • Welcome and Introduction: Professor Norris and Linda introduce the podcast and the topic of the week, discussing the general concept of representation degeneration in neural language models. • The Culprit: Rare Tokens: Linda explains the paper's core empirical finding: rare token embeddings degenerate first and drag the rest of the tokens into a narrow cone. • Adaptive Gradient...

In this episode: • Introduction: The Gold Standard in Question: Professor Norris and Linda introduce the episode's paper from DeepMind, setting the stage by defining perplexity and explaining why it is universally used to evaluate language models. • The Copy Task and the Illusion of Confidence: Linda breaks down the theoretical proof using a bitstring copy task, explaining how high confidence in a...

In this episode: • Introduction to Downstream Scaling Laws: Linda and Professor Norris introduce the paper and discuss the limitations of traditional parametric scaling laws for predicting downstream task performance. • The Token-Level Secret: Linda explains how NeuNeu uses token-level probabilities instead of average validation loss to capture critical distributional signals. • Architecture Deep...

In this episode: • Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models. • Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers. •...

In this episode: • Welcome & Introduction: Professor Norris and Linda welcome the listeners. Linda introduces the paper of the week, teasing the unexpected comeback of non-linear RNNs. • The Expressivity Gap: Linear vs. Non-Linear RNNs: The hosts discuss how linear RNNs like Mamba and Gated DeltaNet dominate due to their efficiency, but fundamentally lack the expressive power for complex state-tra...

In this episode: • Chapter 1: Introduction to the Bottleneck: Linda introduces the paper and the general concept of the LM head. Professor Norris expresses initial skepticism about revisiting the softmax bottleneck. • Chapter 2: Expressivity vs. Optimization: The hosts discuss how the paper shifts the focus from the classical expressivity limitation to a fundamental optimization problem. • Chapter...

In this episode: • The Context Window Illusion: Norris and Linda introduce the episode and the paper, discussing why million-token context windows don't automatically solve reasoning tasks. • The Math of Score Dilution: Linda dives into the theoretical bottleneck of static self-attention, explaining why the target-distractor margin must scale logarithmically. • Query-Only Test-Time Training: Linda...

In this episode: • Introduction to State-Tracking: Linda and Professor Norris introduce the paper and discuss the historical context of state-tracking in sequence models. • The Next-Token Prediction Testbed: The hosts discuss how the authors used Python REPL traces with print statements to evaluate models using next-token prediction instead of sequence-to-sequence. • DeltaNet Triumphs Over Transfo...

In this episode: • Dessert Before Vegetables?: Professor Norris and Linda introduce the concept of Curriculum Learning in LLMs and discuss why the intuitive idea of saving the best data for last has historically failed to produce significant results. • The Invisible antagonist: Learning Rate Decay: Linda reveals the paper's core insight: standard learning rate schedules decay to near-zero just as...

In this episode: • Welcome & The End of Vibe Coding?: Linda introduces GLM-5 and the paradigm shift from passive vibe coding to autonomous agentic engineering. • Architecture & DeepSeek Sparse Attention: Professor Norris and Linda examine the 744B parameter model and how transitioning from dense to sparse attention drastically cuts compute costs. • Asynchronous RL and the Slime Framework: A deep d...

In this episode: • Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism. • The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function. • The One-Line Fix and S...

In this episode: • Welcome and Introduction: Professor Norris and Linda introduce the episode and the paper of the week: 'Backward Gradient Normalization in Deep Neural Networks'. • The Ghost of Gradients Past: A discussion on the classic vanishing and exploding gradient problems, and why existing solutions like Batch Normalization and ResNets still leave room for improvement. • Unpacking Backward...

In this episode: • The PreNorm Dilution Problem: Professor Norris and Linda introduce the episode and discuss the fundamental limitations of standard residual connections, focusing on the unbounded magnitude growth caused by PreNorm. • Attention Residuals and the Time-Depth Duality: Linda introduces the core concept of Full Attention Residuals, treating network depth like sequence length. Professo...

In this episode: • Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper. • The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information. • Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where. • Empirical Triumphs: Reviewing t...

In this episode: • Introduction to Precision in Scaling Laws: Linda introduces the new paper which adds precision as a third variable to the Chinchilla scaling laws. Professor Norris reflects on how precision is usually treated as an afterthought. • The Post-Training Quantization Paradox: The hosts discuss the surprising finding that overtraining models on too much data actually makes them degrade...