Mechanical Dreams

In this episode: • A Picture is Worth a Thousand Tokens: The hosts introduce the challenge of long context in LLMs and present the paper's radical idea: compressing text by taking a picture of it. • Compressing Text into Pixels: A deep dive into the main concept of optical compression, exploring how a page of text can be represented with far fewer vision tokens than text tokens. • The Secret Sauce...

In this episode: • The Need for Speed: Microscaling Formats: Linda introduces new low-precision MX formats for training LLMs, designed to save massive amounts of compute. Professor Norris is intrigued but skeptical about the practical trade-offs. • When Good Training Goes Bad: The hosts discuss the core problem identified in the paper: severe training instabilities and sudden, unrecoverable loss s...

In this episode: • The Synthetic Data Gold Rush: The hosts introduce the data scarcity problem for training large language models and present today's paper, which systematically investigates synthetic data as a potential solution. • Real Fake Data: What Kinds Are We Talking About?: Linda breaks down the different types of synthetic data studied, including rephrased web text and entirely novel 'syn...

In this episode: • Introduction: Less is More in Optimization?: Professor Norris and Linda introduce the "Drop-Muon" paper, which challenges the fundamental assumption that all neural network layers must be updated at every training step. They set the stage by questioning if selectively updating layers could lead to faster convergence. • A Refresher on the Muon Family: Linda provides a high-level...

In this episode: • The Finicky Diet of Large Language Models: Linda introduces a paper about how LLMs learn from mixtures of web data and high-quality data. Professor Norris expresses his initial intuition that more data is always better, setting the stage for the paper's surprising findings. • It's Not a Slope, It's a Cliff: Unveiling Phase Transitions: The hosts discuss the paper's core finding:...

In this episode: • Another Week, Another 'Open' Model?: Linda introduces the Apertus paper, framing it as a response to the systemic shortcomings of current open models. Professor Norris questions what makes this one different from the countless other 'open' releases. • Data Compliance and the Goldfish in the Machine: The hosts dive into Apertus's strict data compliance, including its novel retroa...

In this episode: • The Forgetful Student: Professor Norris and Linda introduce the central problem: Large Language Models often struggle to reliably learn and recall facts. They set the stage for this week's paper, which proposes a solution inspired by how humans study. • Learning by Self-Teaching: Linda explains the core concept of 'Active Reading,' where a model generates its own diverse study m...

In this episode: • The Optimizer Royal Rumble: Professor Norris and Linda introduce the chaotic landscape of LLM optimizers, where everyone claims to beat the reigning champion, AdamW. They introduce today's paper, which aims to be the referee in this messy fight. • The Art of the Unfair Comparison: Linda explains the paper's core thesis: many new optimizers seem fast only because they are compare...

In this episode: • Beyond Adam: The Great Optimizer Bake-Off: Linda introduces a paper questioning the decade-long reign of the AdamW optimizer for training large language models. Professor Norris expresses his healthy skepticism about the endless stream of 'new and improved' optimizers. • Adam's Kingdom and Its Challengers: The hosts discuss why AdamW became the default and the paper's motivation...

In this episode: • Opening Up a New Chapter: Apertus: Linda introduces the Apertus paper, highlighting its focus on data compliance and extreme multilingualism. Professor Norris is intrigued but skeptical about what 'fully open' and 'compliant' truly mean in practice. • Clean Data, Clear Conscience?: The hosts discuss Apertus's novel approach to data compliance, including retroactively honoring ro...

In this episode: • Introduction: More Than an Improv Artist? • The Two Faces of Prediction: Linda breaks down the key difference between teacher-forced training and autoregressive inference. Professor Norris likens this to learning to drive with an instructor who constantly corrects you, which doesn't prepare you for driving alone. • The Clever Hans Cheat: Linda explains the paper's core concept:...

In this episode: • More Than Just NPCs: Linda introduces this week's paper, a comprehensive survey on the roles of Large Language Models in games. Professor Norris provides some witty historical context on game AI, setting the stage for the discussion. • What Counts as 'Large'?: The hosts discuss the paper's specific definition of a Large Language Model. They cover why establishing a clear scope,...

In this episode: • The Benchmark Treadmill: Linda introduces the problem with existing ML benchmarks, noting they are often either too easy or too artificial. Professor Norris adds witty commentary on how quickly new models seem to 'solve' and saturate these tests. • Let's Ask the Unanswerable: Linda presents the core idea from the UQ paper: evaluating models on genuinely unsolved questions from p...

In this episode: • The Billion-Dollar Guessing Game: Professor Norris and Linda introduce the high-stakes problem of LLM evaluation. Linda presents today's paper, which offers a framework to make our small-scale experiments more predictive of large-scale success. • Tuning In the Signal, Tuning Out the Noise: Linda breaks down the paper's core concepts: 'signal' as a benchmark's ability to distingu...

In this episode: • Can a Jack-of-All-Trades Learn to Be a Doctor?: Linda introduces the challenge of specializing large language models for specific domains and presents a new paper that proposes a smarter way to pick the right training data. • The Deceptive First Sip: The hosts discuss why the common practice of 'micro-annealing'—using small-scale tests to evaluate data sources—can be misleading,...

In this episode: • The Transformer's Black Box: Linda introduces the 'Thinking Like Transformers' paper, highlighting the challenge of understanding the computational model behind transformers, unlike RNNs and their connection to finite state machines. Professor Norris agrees, sharing a witty remark about the opacity of modern deep learning models. • Introducing RASP: A Language for Transformers:...

In this episode: • Honey, I Shrunk the Activated Parameters: Linda introduces the massive Kimi K2 paper, focusing on its 'agentic intelligence' and surprisingly small number of activated parameters. Professor Norris offers his initial witty skepticism about yet another trillion-parameter model. • Taming the Exploding Logits: The hosts get technical, discussing the novel MuonClip optimizer designed...

In this episode: • A New ERNIE on the Block: Linda introduces the new ERNIE 4.5 technical report from Baidu, setting the stage for a discussion on their new family of large-scale foundation models, including their massive 424 billion parameter Mixture-of-Experts model. • Not Your Average MoE: The hosts discuss the core concept of ERNIE 4.5: its Mixture-of-Experts (MoE) architecture. Linda explains...