uk

Mechanical Dirk

Mechanical Dreams

Science EN ↓ Епізодів: 181

An automatically generated podcast about machine learning and natural language processing. The two fictional hosts talk about papers that I want to learn more about on my way to work. It's not good, but it's useful.

Обов'язково відвідайте сайт подкасту та підтримайте його автора: mechanicaldreams.mechanicaldirk.com

Автор

Mechanical Dirk

Категорія

Science

Сайт подкасту

mechanicaldreams.mechanicaldirk.com

Останній епізод

12 лис 2026

Де слухати?

Подкасти в застосунку Replaio Radio Уже незабаром

Подкасти незабаром з'являться в застосунку. Встановіть уже зараз і першими побачте зовсім новий погляд на подкасти

Завантажити з Google Play Встановіть безкоштовно Android 10 млн+ завантажень · рейтинг 4,7+ iOS незабаром

Епізоди

You don’t need long context 12.11.2026

In this episode: • The Context Window Arms Race: Professor Norris and Linda introduce the episode and discuss a new NVIDIA paper challenging the necessity of massive context windows, highlighting the hidden computational and monetary costs. • Long-Horizon Tasks and End-of-Conversation Compaction: Linda explains End-of-Conversation Compaction (EoCC) and how language models can tackle long-horizon s...

Scaling Hierarchical Networks 11.11.2026

In this episode: • Intro and The Bitter Lesson: Introducing the podcast and the Cartesia paper on Hierarchical Networks. • Scaling Ladders and The BPE Comparison: Discussing the compute efficiency gains of 1-stage H-Nets over traditional BPE transformers. • The 3 Billion Parameter Matchup: Comparing a 3.67B parameter H-Net to SmolLM3 and analyzing the 10x data efficiency. • In-Context Tokenization...

Perturbing Inputs Improves Pretraining Distillation 10.11.2026

In this episode: • Introduction and Standard Distillation: Professor Norris and Linda introduce the podcast and set the baseline for standard knowledge distillation in language models. • The Core Idea of Perturbed Inputs: Linda introduces the concept of adding noise to the teacher and student inputs, addressing Professor Norris's initial skepticism. • Three Flavors of Perturbation: A deep dive int...

Why is AI-generated language so hard to understand 09.11.2026

In this episode: • Introduction: The Reader-Patience Budget: Professor Norris and Linda introduce the episode's topic, discussing the growing sentiment that AI-generated text is becoming overly convoluted. • The Linguistic Culprits of Complexity: Linda breaks down the specific psycholinguistic features identified in the paper, such as denominal verbs and stacked nouns, that increase human cognitiv...

Muse Glimmer 30B 08.11.2026

In this episode: • Introduction & Paper Overview: Professor Norris and Linda introduce the episode and the Muse Glimmer 30B paper, outlining its goal of bringing agentic capabilities to consumer hardware. • Architecture & The Dense vs MoE Debate: Linda explains the 30B dense architecture and why the authors chose it over Mixture-of-Experts for strict 32GB memory constraints. • Distillation & Agent...

HyScale 07.11.2026

In this episode: • The Context Bottleneck: Professor Norris and Linda introduce the paper HyScale, discussing the problem with long-context training and how learned sparse attention works. • The Key Insight: Indices, Not Activations: They dive into the core bottleneck of sparse attention, which is the selection step, and reveal the paper's key insight about discrete indices. • HyTopK: On-the-Fly S...

Forking 06.11.2026

In this episode: • Introduction to Forking: Linda introduces the concept of forking and the paper from MetaCircle Research. • The Autoresearch Discovery and the Anatomy of Forking: Discussion on how an automated AI agent discovered forking and what the phenomenon looks like. • Mechanisms: N-gram Memory and Context Subspaces: Deep dive into over-encoding, n-gram lookup tables, and how they cause cr...

Exclusive Self Attention 05.11.2026

In this episode: • Welcome and Paper Introduction: Professor Norris and Linda introduce the podcast and today's paper on Exclusive Self Attention by Shuangfei Zhai. • The Problem of Attention Similarity Bias: Linda explains how standard self-attention redundantly focuses on the token's own value vector instead of context. • The Elegant Solution of XSA: The hosts discuss the mathematical solution,...

Don't Drop Dropout 04.11.2026

In this episode: • The Dropout Paradox: Introduction to the paper and the historical context of abandoning dropout in large language models. • Granularity and Tensor Tricks: Discussion on layer versus sub-layer dropout and the benefits of per-sequence application. • The Magic Recipe ILD and DTS: Breaking down the optimal configurations of Increasing Layer Distribution and Decreasing Time Schedule....

Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation 03.11.2026

In this episode: • Introduction and the Long Generation Bottleneck: Linda introduces the SambaY paper and discusses the memory I/O bottleneck in generating long Chain-of-Thought reasoning. • SambaY and Gated Memory Units: A deep dive into the decoder-hybrid-decoder architecture and how Gated Memory Units replace expensive cross-attention layers. • Scaling Laws and Rigorous Comparisons: Discussion...

Before They Can Solve 03.11.2026

In this episode: • Welcome & The Base Model Dilemma: Professor Norris and Linda introduce the problem of evaluating base models for expensive post-training. • The Goldilocks Problem of Benchmarks: The hosts discuss why existing evaluations are either too complex or not representative enough to predict agentic capability. • The Decisive Step Methodology: Linda explains how the researchers use succe...

Qwen3.8-Next 27.09.2026

In this episode: • Welcome and the 1/9th FLOP Miracle: Linda introduces Qwen3.8-Next and its massive efficiency gains, hooking Dr. Norris. • Token Mixing with GDN and Sparse Attention: Discussing the Gated DeltaNet hybrid architecture and how Qwen Sparse Attention scales to long contexts. • The Gated Residual (GR): Norris is skeptical about widening the residual stream, but Linda explains the gati...

Distillation Scaling Laws 26.09.2026

In this episode: • Introduction to Distillation and Inference Costs: Linda and Norris introduce the paper and discuss why compute-optimal models are driving the need for smaller, distilled models to reduce inference costs. • The Distillation Scaling Law and the Capacity Gap: The hosts break down the empirical scaling law discovered in the paper and explore the counter-intuitive capacity gap where...

Decode-Branch Transformers 25.09.2026

In this episode: • The Inference Bottleneck: Introduction to the podcast and the fundamental asymmetry between compute-bound prefill and memory-bound decoding in LLMs. • Enter the Decode-Branch: Linda introduces the Decode-Branch Transformer, explaining how it separates prompt processing from continuation prediction without duplicating the KV cache. • Asymmetric Attention & Shared Weights: A deep...

The State-Prediction Separation Hypothesis 23.08.2026

In this episode: • Introduction to State-Prediction Separation: Professor Norris and Linda introduce the paper and the core problem of standard Transformers conflating next-token prediction and future state representation. • The SPS Mechanism: Linda explains how the authors introduce a dummy predict token to separate the input state stream from the prediction stream. • Experimental Results and Bas...

Pretraining Mixture of Experts for Emergent Modularity 22.08.2026

In this episode: • Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models. • The Document-Level Routing Constraint: Linda explains EMOs core mechanism of restricting tokens in a document to a shared pool of experts. • Overcoming Training Instability: The hosts discuss the conflict with local load bala...

Muon-Optimized Distillation and Quantization for LLM Deployment 21.08.2026

In this episode: • Introduction & The Edge Deployment Problem: Professor Norris and Linda introduce the episode's paper, focusing on the difficulty of running large language models on edge devices with limited memory and compute. • The Multi-Teacher Distillation Pipeline: Linda explains the dual-teacher approach, using Llama 4 Scout for synthetic data generation and Llama 3.3 70B for logit-based k...

Continuous-Query Limited Memory Language Models 20.08.2026

In this episode: • Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in model weights. • From Relational to Continuous Queries: Linda explains the limitations of prior Limited Memory Language Models (LMLMs) and introduces Co-LMLM's continuous-query approach. • Under the Hood: Training and The Toke...

QuaRot 08.08.2026

In this episode: • Introduction to Quantization and Outliers: Norris and Linda introduce the episode's paper, QuaRot, and discuss the main hurdle in LLM inference: the memory bottleneck and the pesky outlier features in activations. • The Magic of Hadamard Rotations: Linda explains the core mechanism of QuaRot, using randomized Hadamard transformations to eliminate outliers through computational i...

How to set AdamW’s weight decay as you scale model and dataset size 07.08.2026

In this episode: • Introduction to Mechanical Dreams and the AdamW Scaling Problem: Professor Norris and Linda introduce the podcast and discuss the costly problem of hyperparameter tuning for large-scale machine learning models. • Reframing AdamW as an Exponential Moving Average: Linda reveals the paper's core insight that AdamW can be mathematically understood as an EMA of recent weight updates....

Weight Decay Improves Language Model Plasticity 07.08.2026

In this episode: • Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity. • Challenging the Validation Loss Paradigm: Linda explains how optimizing for pretraining validation loss can actually hurt downstream fine-tuning performance. • The Mechanics of Plasticity: A deep dive into why weight de...

Bridging Compute- and Data-Optimal Pretraining 30.07.2026

In this episode: • The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling. • The Compute-Data Scaling Law: Linda introduces the new CD-scaling law, focusing on the token effectiveness function that bridges compute-optimal and data-optimal scaling. • Eta and the Saturation Ceiling: A deep dive into how dat...

Pushing LLM Pretraining Scales 29.07.2026

In this episode: • Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW. • The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models. • Muon and Spectral Descent: Linda explains how Muon...

Why Larger Models Learn More 16.07.2026

In this episode: • Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot, challenging the standard sample-efficiency dogma. • The Statistical Bottleneck and Task Utility: Linda explains the paper's synthetic regression setup, detailing how models prioritize tasks based on a utility metric derived from...

Rethinking the Role of Efficient Attention in Hybrid Architectures 15.07.2026

In this episode: • Introduction to Hybrid Architectures: Professor Norris and Linda introduce the episode's paper on hybrid attention architectures and discuss why scaling full attention is computationally expensive. • The Scaling Surprise: Linda reveals that different efficient attention designs eventually converge to the same long-context performance, surprising Dr. Norris. • Large-Window Lazine...

Слухайте подкаст Mechanical Dreams у Replaio

Радіо та подкасти в одному застосунку - безкоштовно й без реєстрації. Встановіть уже сьогодні та не пропустіть запуск

Завантажити з Google Play

Replaio не є видавцем подкастів; назви шоу, обкладинки та аудіо належать їхнім авторам і поширюються через публічні RSS-канали