Mechanical Dreams

In this episode: • Is Query Redundant?: Linda introduces a provocative paper suggesting a core part of the Transformer attention mechanism, the Query matrix, might be unnecessary. Professor Norris expresses his trademark skepticism about simplifying such a fundamental component. • The Usual Suspects: Q, K, and V: Linda provides a quick, intuitive refresher on the roles of Query, Key, and Value mat...

In this episode: • Why Does Seed 42 Work Best?: Linda introduces a paper that tries to answer a classic machine learning question: why does the random seed have such a big impact on training? Professor Norris laments that this is a problem as old as neural networks themselves. • A Roadmap for Training: Linda explains the paper's novel approach of using a Hidden Markov Model to turn messy training...

In this episode: • Why a Good Pre-trainer Isn't Always a Good Finetuner: The hosts introduce the puzzle of pre-training: why doesn't a lower cross-entropy loss always guarantee better performance after fine-tuning? They set the stage for today's paper which proposes a new perspective. • Are We Covering Our Bases? The Coverage Principle: Linda explains the paper's central concept of 'coverage,' a m...

In this episode: • The Art and Science of Scaling RL: Professor Norris and Linda introduce today's topic, a new paper from Meta that aims to make training large models with reinforcement learning more predictable and scientific. • More Art than Science: Linda explains why scaling Reinforcement Learning is so difficult compared to pre-training, highlighting the lack of predictive scaling laws and t...

In this episode: • The Frozen Brains of AI: Linda introduces the problem of static LLMs and the challenge of 'catastrophic forgetting.' Professor Norris provides historical context on this long-standing issue in AI and introduces the day's paper on continual learning. • Why Can't Models Just Keep Learning?: The hosts discuss traditional approaches to continual learning, like data replay and regula...

In this episode: • A Picture is Worth a Thousand Tokens: The hosts introduce the challenge of long context in LLMs and present the paper's radical idea: compressing text by taking a picture of it. • Compressing Text into Pixels: A deep dive into the main concept of optical compression, exploring how a page of text can be represented with far fewer vision tokens than text tokens. • The Secret Sauce...

In this episode: • The Need for Speed: Microscaling Formats: Linda introduces new low-precision MX formats for training LLMs, designed to save massive amounts of compute. Professor Norris is intrigued but skeptical about the practical trade-offs. • When Good Training Goes Bad: The hosts discuss the core problem identified in the paper: severe training instabilities and sudden, unrecoverable loss s...

In this episode: • The Synthetic Data Gold Rush: The hosts introduce the data scarcity problem for training large language models and present today's paper, which systematically investigates synthetic data as a potential solution. • Real Fake Data: What Kinds Are We Talking About?: Linda breaks down the different types of synthetic data studied, including rephrased web text and entirely novel 'syn...

In this episode: • Introduction: Less is More in Optimization?: Professor Norris and Linda introduce the "Drop-Muon" paper, which challenges the fundamental assumption that all neural network layers must be updated at every training step. They set the stage by questioning if selectively updating layers could lead to faster convergence. • A Refresher on the Muon Family: Linda provides a high-level...

In this episode: • The Finicky Diet of Large Language Models: Linda introduces a paper about how LLMs learn from mixtures of web data and high-quality data. Professor Norris expresses his initial intuition that more data is always better, setting the stage for the paper's surprising findings. • It's Not a Slope, It's a Cliff: Unveiling Phase Transitions: The hosts discuss the paper's core finding:...

In this episode: • Another Week, Another 'Open' Model?: Linda introduces the Apertus paper, framing it as a response to the systemic shortcomings of current open models. Professor Norris questions what makes this one different from the countless other 'open' releases. • Data Compliance and the Goldfish in the Machine: The hosts dive into Apertus's strict data compliance, including its novel retroa...

In this episode: • The Forgetful Student: Professor Norris and Linda introduce the central problem: Large Language Models often struggle to reliably learn and recall facts. They set the stage for this week's paper, which proposes a solution inspired by how humans study. • Learning by Self-Teaching: Linda explains the core concept of 'Active Reading,' where a model generates its own diverse study m...

In this episode: • The Optimizer Royal Rumble: Professor Norris and Linda introduce the chaotic landscape of LLM optimizers, where everyone claims to beat the reigning champion, AdamW. They introduce today's paper, which aims to be the referee in this messy fight. • The Art of the Unfair Comparison: Linda explains the paper's core thesis: many new optimizers seem fast only because they are compare...

In this episode: • Beyond Adam: The Great Optimizer Bake-Off: Linda introduces a paper questioning the decade-long reign of the AdamW optimizer for training large language models. Professor Norris expresses his healthy skepticism about the endless stream of 'new and improved' optimizers. • Adam's Kingdom and Its Challengers: The hosts discuss why AdamW became the default and the paper's motivation...

In this episode: • Opening Up a New Chapter: Apertus: Linda introduces the Apertus paper, highlighting its focus on data compliance and extreme multilingualism. Professor Norris is intrigued but skeptical about what 'fully open' and 'compliant' truly mean in practice. • Clean Data, Clear Conscience?: The hosts discuss Apertus's novel approach to data compliance, including retroactively honoring ro...

In this episode: • Introduction: More Than an Improv Artist? • The Two Faces of Prediction: Linda breaks down the key difference between teacher-forced training and autoregressive inference. Professor Norris likens this to learning to drive with an instructor who constantly corrects you, which doesn't prepare you for driving alone. • The Clever Hans Cheat: Linda explains the paper's core concept:...

In this episode: • More Than Just NPCs: Linda introduces this week's paper, a comprehensive survey on the roles of Large Language Models in games. Professor Norris provides some witty historical context on game AI, setting the stage for the discussion. • What Counts as 'Large'?: The hosts discuss the paper's specific definition of a Large Language Model. They cover why establishing a clear scope,...

In this episode: • The Benchmark Treadmill: Linda introduces the problem with existing ML benchmarks, noting they are often either too easy or too artificial. Professor Norris adds witty commentary on how quickly new models seem to 'solve' and saturate these tests. • Let's Ask the Unanswerable: Linda presents the core idea from the UQ paper: evaluating models on genuinely unsolved questions from p...