Mechanical Dreams

In this episode: • Welcome and the 1/9th FLOP Miracle: Linda introduces Qwen3.8-Next and its massive efficiency gains, hooking Dr. Norris. • Token Mixing with GDN and Sparse Attention: Discussing the Gated DeltaNet hybrid architecture and how Qwen Sparse Attention scales to long contexts. • The Gated Residual (GR): Norris is skeptical about widening the residual stream, but Linda explains the gati...

In this episode: • Introduction to Distillation and Inference Costs: Linda and Norris introduce the paper and discuss why compute-optimal models are driving the need for smaller, distilled models to reduce inference costs. • The Distillation Scaling Law and the Capacity Gap: The hosts break down the empirical scaling law discovered in the paper and explore the counter-intuitive capacity gap where...

In this episode: • The Inference Bottleneck: Introduction to the podcast and the fundamental asymmetry between compute-bound prefill and memory-bound decoding in LLMs. • Enter the Decode-Branch: Linda introduces the Decode-Branch Transformer, explaining how it separates prompt processing from continuation prediction without duplicating the KV cache. • Asymmetric Attention & Shared Weights: A deep...

In this episode: • Introduction to State-Prediction Separation: Professor Norris and Linda introduce the paper and the core problem of standard Transformers conflating next-token prediction and future state representation. • The SPS Mechanism: Linda explains how the authors introduce a dummy predict token to separate the input state stream from the prediction stream. • Experimental Results and Bas...

In this episode: • Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models. • The Document-Level Routing Constraint: Linda explains EMOs core mechanism of restricting tokens in a document to a shared pool of experts. • Overcoming Training Instability: The hosts discuss the conflict with local load bala...

In this episode: • Introduction & The Edge Deployment Problem: Professor Norris and Linda introduce the episode's paper, focusing on the difficulty of running large language models on edge devices with limited memory and compute. • The Multi-Teacher Distillation Pipeline: Linda explains the dual-teacher approach, using Llama 4 Scout for synthetic data generation and Llama 3.3 70B for logit-based k...

In this episode: • Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in model weights. • From Relational to Continuous Queries: Linda explains the limitations of prior Limited Memory Language Models (LMLMs) and introduces Co-LMLM's continuous-query approach. • Under the Hood: Training and The Toke...

In this episode: • Introduction to Quantization and Outliers: Norris and Linda introduce the episode's paper, QuaRot, and discuss the main hurdle in LLM inference: the memory bottleneck and the pesky outlier features in activations. • The Magic of Hadamard Rotations: Linda explains the core mechanism of QuaRot, using randomized Hadamard transformations to eliminate outliers through computational i...

In this episode: • Introduction to Mechanical Dreams and the AdamW Scaling Problem: Professor Norris and Linda introduce the podcast and discuss the costly problem of hyperparameter tuning for large-scale machine learning models. • Reframing AdamW as an Exponential Moving Average: Linda reveals the paper's core insight that AdamW can be mathematically understood as an EMA of recent weight updates....

In this episode: • Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity. • Challenging the Validation Loss Paradigm: Linda explains how optimizing for pretraining validation loss can actually hurt downstream fine-tuning performance. • The Mechanics of Plasticity: A deep dive into why weight de...

In this episode: • The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling. • The Compute-Data Scaling Law: Linda introduces the new CD-scaling law, focusing on the token effectiveness function that bridges compute-optimal and data-optimal scaling. • Eta and the Saturation Ceiling: A deep dive into how dat...

In this episode: • Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW. • The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models. • Muon and Spectral Descent: Linda explains how Muon...

In this episode: • Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot, challenging the standard sample-efficiency dogma. • The Statistical Bottleneck and Task Utility: Linda explains the paper's synthetic regression setup, detailing how models prioritize tasks based on a utility metric derived from...

In this episode: • Introduction to Hybrid Architectures: Professor Norris and Linda introduce the episode's paper on hybrid attention architectures and discuss why scaling full attention is computationally expensive. • The Scaling Surprise: Linda reveals that different efficient attention designs eventually converge to the same long-context performance, surprising Dr. Norris. • Large-Window Lazine...

In this episode: • Welcome and the Autoregressive Bottleneck: Linda introduces the Nemotron-Labs-Diffusion paper and the hosts discuss the fundamental limits of sequential decoding. • The Best of Both Worlds: Joint Training: The hosts dive into how combining Autoregressive and Diffusion objectives creates a complementary, rather than competing, dynamic. • Drafting with Diffusion: Self-Speculation:...

In this episode: • The Memory Wall and the 90 Percent Waste: Professor Norris and Linda introduce the episode's paper, discussing the GPU memory bottleneck caused by ultra-long context windows and the surprising observation about wasted Key-Value cache. • Lookahead Sparse Attention: Linda explains the core concept of LSA, where the model proactively predicts and fetches only the critical context c...

In this episode: • Distillation vs. Reinforcement Learning: Linda and Norris introduce the paper and discuss the differences between off-policy and on-policy distillation. • The Math Behind the Magic: The hosts dive into the mathematical equivalence between on-policy distillation and KL-constrained dense RL. • Extrapolating the Reward: Linda introduces Generalized OPD and the concept of reward ext...

In this episode: • Welcome and the Over-smoothing Problem: Professor Norris and Linda introduce the episode and discuss how deep Transformers suffer from over-smoothing, losing initial token-level information in later layers. • Introducing ResFormer and Value Residuals: Linda explains the core mechanism of ResFormer, which adds a residual connection specifically to the Value vectors from the first...

In this episode: • Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting. • Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure featu...

In this episode: • Introduction to the Compute Divide: Linda and Professor Norris introduce the podcast and discuss the massive computational barriers in modern LLM pretraining before introducing the HRM-Text paper. • Biological Inspiration and the HRM Architecture: The hosts discuss how the human brain's frontoparietal loop inspired the dual-timescale Hierarchical Recurrent Model, breaking down t...

In this episode: • Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice. • Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to toler...

In this episode: • Introduction to the Edge of Stability: Professor Norris and Linda introduce the paper and the surprising behavior of full-batch gradient descent. • Progressive Sharpening: Linda explains how gradient descent naturally navigates towards steeper areas of the loss landscape, increasing sharpness. • Surviving the Edge: The hosts discuss how neural networks avoid catastrophic diverge...

In this episode: • Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models. • The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activati...

In this episode: • Welcome & The Quest for Lifetime Memory: Linda introduces the paper on Memory Sparse Attention (MSA) and sets the stage by comparing current LLM context windows to human lifelong memory capacity. • The Context Length Bottleneck: Professor Norris and Linda discuss why current approaches like full attention, fixed-size memory states (RNNs), and traditional RAG systems struggle to...

In this episode: • Welcome and the Post-hoc Problem: Linda introduces the paper and the hosts discuss why post-hoc unlearning methods fall short against adversarial attacks. • Token vs. Document Filtering: An exploration of why token-level filtering acts as a scalpel compared to the blunt instrument of document filtering. • Scaling Labels with SAEs: The hosts discuss how the authors use Sparse Aut...