AI Post Transformers

This episode explores AgenticQwen, a system for training small open-weight language models to handle industrial-scale tool use through repeated reinforcement learning and synthetic data generation. It explains the paper’s central idea of dual data flywheels: one that turns reasoning failures into harder verifiable tasks, and another that expands simple agent workflows into branching, tool-using tr...

This episode explores a 2025 DeepMind and University of Alberta preprint arguing that AI is reaching the limits of learning from human-generated data and that the next major advances will come from agents learning through interaction and feedback in environments. It explains the shift from static pretraining to grounded reinforcement learning, defining key ideas like long-term reward optimization,...

This episode explores Leo Breiman’s “Statistical Modeling: The Two Cultures” as a sharp argument about a core divide in data science: building explicit probabilistic models to explain the world versus training algorithms that simply predict well on new data. It traces how that split maps onto inference versus prediction, connects Breiman’s critique to earlier ideas from Tukey and Box, and shows ho...

This episode explores Moonshot AI’s KIMI K2.5, a 2026 multimodal model that aims to improve text reasoning, visual understanding, and agent-style task execution within a single open system. It explains the paper’s two main bets: training text and vision jointly from early stages rather than bolting vision on later, and using an external “agent swarm” orchestration layer to split wide-search tasks...

This episode explores ScoutAttention, a systems paper on speeding up long-context LLM inference by managing KV cache growth more intelligently instead of treating it as a pure memory-capacity problem. It explains why long prompts, retrieval-heavy inputs, and extended reasoning traces make decoding increasingly constrained by memory traffic, and why simply offloading cache data to CPU memory can st...

This episode explores DeepSeek-V4’s claim that million-token context windows may finally be practical for real-world use, not just benchmark demos. It explains how the model combines hybrid attention, mixture-of-experts routing, and manifold-constrained hyper-connections to reduce the usual memory and compute costs of long-context transformers while trying to preserve reasoning, coding, and agent...

This episode explores whether the Muon optimizer can truly scale from promising small experiments to frontier-style language model training, and what it would mean if its reported roughly 2x compute-efficiency gain over AdamW holds up. It examines how Muon differs from standard optimizer setups by applying momentum plus orthogonalization to matrix-shaped hidden-layer weights while keeping embeddin...

This episode explores a Princeton paper on whether multiple long-running, tool-using AI agent trajectories can be combined more effectively by an “aggregator agent” that selectively inspects the full traces, rather than by simple answer voting or compressed summaries. It explains why aggregation gets much harder for long-horizon agentic tasks like web research, navigation, and software repair, whe...

This episode explores TokenDance, a systems approach for serving many LLM-based agents more efficiently by collectively sharing transformer KV caches across synchronized conversation rounds. It explains why multi-agent workloads are fundamentally different from ordinary chat serving: agents persist across rounds, accumulate large KV caches, and often follow an “all-gather” pattern where each agent...

This episode explores RetrievalAttention, a 2024 paper that tries to make long-context LLM inference much cheaper by retrieving only the most relevant key-value cache entries during decoding instead of scanning the entire history every time. It explains why long-context serving is bottlenecked less by raw FLOPs than by memory traffic and KV-cache growth, citing concrete figures such as roughly 125...

This episode explores a 2023 paper on Deep Spiking Q-Networks, asking whether a directly trained spiking version of DQN can compete with earlier conversion-based spiking reinforcement learning methods on Atari while retaining the energy-efficiency promise of spiking neural networks. It explains the technical foundations behind spiking networks, including leaky integrate-and-fire neurons, surrogate...

This episode explores DreamerV3, a world-model reinforcement learning system that claims to use one main configuration across more than 150 tasks spanning Atari, ProcGen, DMLab, robot control, visual control, BSuite, and Minecraft. It explains how world models work—learning compact environment dynamics so an agent can train on imagined futures—and why that approach is appealing for sample efficien...

This episode explores a 2021 Google Research paper on whether large language models can synthesize short Python programs directly from natural-language descriptions, moving beyond code autocomplete into true program synthesis. It explains why this is difficult in general-purpose languages, contrasts classical search-based synthesis with transformer-based generation, and highlights the paper’s emph...

This episode explores whether multi-agent systems can benefit from test-time scaling in the same way single models do, focusing on a 2025 paper that combines learned collaborative reasoning with runtime orchestration. It explains the paper’s core setup: a model trained on 500 carefully curated multi-agent reasoning traces (M500) and a separate “CEO” controller that coordinates specialized agents s...

This episode explores TUMIX, a test-time scaling framework that turns a single strong language model into a team of specialized agents with different tool-use strategies, including plain-text reasoning, code execution, search, and hybrids. It explains the paper’s core argument that better reasoning may come not from simply sampling one model more times, but from diversifying computational pathways...

This episode explores a paper that tests whether general LLM agents remain effective when search, coding, reasoning, and API/tool-use tasks are mixed together under one shared prompt, interface, and tool set rather than optimized benchmark-specific setups. It explains how the benchmark is built by unifying tasks from BrowseComp, WebVoyager, SWE-Bench Verified, Terminal-Bench, MathHay, Tau2-Bench,...

This episode explores a 2026 paper on AgentArk, which asks whether the reasoning gains of multi-agent LLM systems can be compressed into a single model, reducing the latency, token cost, and orchestration burden of running a “committee” of models at inference time. It explains multi-agent systems as setups where multiple model instances debate, critique, and revise one another, arguing that their...

This episode explores a systems paper on making multi-agent LLM setups far more efficient by sharing most of the KV cache across agents that use the same base model with different LoRA adapters. It explains the core argument: for a shared long context, the backbone model’s hidden states are nearly identical across agents, while most role-specific differences come from LoRA’s low-rank adapter outpu...

This episode explores a systems paper on speeding up retrieval-augmented generation by reusing KV caches for frequently repeated retrieved documents, even when those documents are not exact prompt prefixes. It explains why long RAG prompts make prefill the main latency bottleneck, why standard prefix caching only helps in narrow cases, and why naive non-prefix cache reuse can hurt quality by ignor...

This episode explores the Qwen3.5-Omni technical report as a significant update in omnimodal AI: a system designed to understand and generate text, audio, images, and video within one architecture. It unpacks the model’s Thinker–Talker design, arguing that separating multimodal reasoning from real-time output is especially important for speech, where latency and timing make generation far harder t...

This episode explores Gated Delta Networks, a sequence-modeling approach that combines Mamba-style gating with DeltaNet-style selective memory updates to improve long-context and retrieval-heavy performance. It explains how linear attention and state-space models compress the past into a fixed recurrent state, why that makes them hardware-efficient, and where they often fail: memory collisions tha...

This episode explores whether newer hybrid-attention language models make prefill-decode disaggregation practical across clusters or even datacenters by shrinking the KV cache enough to move it over ordinary Ethernet. It explains why the real production bottleneck is not request routing but transferring the attention state between prefill and decode, and contrasts dense transformers—where KV cache...

This episode explores a philosophical challenge to computational functionalism through Alexander Lerchner’s “The Abstraction Fallacy,” which argues that software can simulate conscious behavior without ever producing real subjective experience. It examines whether computation is an objective physical process or an interpretation imposed on physical systems, contrasting Lerchner’s “mapmaker” idea w...

This episode explores NVIDIA’s Nemotron 3 Super, an open 120B-parameter model with only 12B active parameters per token, and examines whether its hybrid Mamba-transformer Mixture-of-Experts design can approach frontier-model accuracy while delivering much better inference efficiency. The discussion breaks down the paper’s main ingredients—MoE sparsity, LatentMoE for more practical serving, Mamba-s...

This episode explores a systems paper on speeding up LLM “Re-Prefill,” the step where a model reloads a previously saved shared prefix KV cache from CPU or SSD, computes a request-specific suffix, and produces the first token. It explains why prefix KV reuse is valuable for shared-context workloads like RAG, conversational search, and multi-turn document QA, but argues that offloading creates new...