Mechanical Dreams

Signal and Noise- A Framework for Reducing Uncertainty in Language Model Evaluation 19.08.2025 14:20 ↓

In this episode: • The Billion-Dollar Guessing Game: Professor Norris and Linda introduce the high-stakes problem of LLM evaluation. Linda presents today's paper, which offers a framework to make our small-scale experiments more predictive of large-scale success. • Tuning In the Signal, Tuning Out the Noise: Linda breaks down the paper's core concepts: 'signal' as a benchmark's ability to distingu...

Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training 18.08.2025 14:43 ↓

In this episode: • Can a Jack-of-All-Trades Learn to Be a Doctor?: Linda introduces the challenge of specializing large language models for specific domains and presents a new paper that proposes a smarter way to pick the right training data. • The Deceptive First Sip: The hosts discuss why the common practice of 'micro-annealing'—using small-scale tests to evaluate data sources—can be misleading,...

Thinking Like Transformers 29.07.2025 13:19 ↓

In this episode: • The Transformer's Black Box: Linda introduces the 'Thinking Like Transformers' paper, highlighting the challenge of understanding the computational model behind transformers, unlike RNNs and their connection to finite state machines. Professor Norris agrees, sharing a witty remark about the opacity of modern deep learning models. • Introducing RASP: A Language for Transformers:...

Kimi K2 28.07.2025 13:29 ↓

In this episode: • Honey, I Shrunk the Activated Parameters: Linda introduces the massive Kimi K2 paper, focusing on its 'agentic intelligence' and surprisingly small number of activated parameters. Professor Norris offers his initial witty skepticism about yet another trillion-parameter model. • Taming the Exploding Logits: The hosts get technical, discussing the novel MuonClip optimizer designed...

ERNIE Technical Report 25.07.2025 11:17 ↓

In this episode: • A New ERNIE on the Block: Linda introduces the new ERNIE 4.5 technical report from Baidu, setting the stage for a discussion on their new family of large-scale foundation models, including their massive 424 billion parameter Mixture-of-Experts model. • Not Your Average MoE: The hosts discuss the core concept of ERNIE 4.5: its Mixture-of-Experts (MoE) architecture. Linda explains...