AI Post Transformers

This episode explores KVzip, a query-agnostic method for compressing long-context KV caches so a model can reuse a shared document, codebase, or memory bank across many later questions without optimizing for just one query. It explains why KV cache has become a major systems bottleneck, including the striking example that a 120,000-token context for Qwen2.5-14B can require more memory for cache th...

This episode explores the DFX system, a four-FPGA appliance designed to accelerate transformer-based text generation by targeting a key weakness of GPUs: low-batch, token-by-token decode. It explains the difference between prompt processing and sequential generation, connects the paper’s older terminology to today’s prefill/decode framing, and shows why autoregressive inference often leaves GPU ha...

This episode explores a paper on inference-time scaling for coding agents, asking whether extra test-time compute still helps when tasks are long, messy, and require multi-step tool use rather than a single code completion. It focuses on the paper’s main argument that the real bottleneck is not generating more rollout attempts, but representing prior attempts well enough to compare, select, and re...

This episode explores Beluga, a systems paper that tackles one of the biggest practical bottlenecks in long-context LLM inference: how to store and retrieve massive KV caches when GPU memory is no longer enough. It explains why traditional RDMA-based memory disaggregation is cumbersome and how Beluga uses CXL-based pooled memory to give GPUs and CPUs more direct, load/store-style access to shared...

This episode explores a systems paper that extends GPU memory through CXL-attached DRAM and SSDs, asking whether accelerators can reach beyond on-board HBM without the usual overhead of software-driven memory migration. It explains CXL, memory disaggregation, and the difference between local GPU memory, host-managed memory, CXL memory, and storage-backed expansion, while grounding the discussion i...

This episode explores Generative Recursive Reasoning, a paper that asks whether models can reason more effectively by repeatedly refining an internal latent state instead of externalizing long chains of thought as tokens. It explains how recursive reasoning trades parameter growth for inference-time computation, and how this approach may be especially useful for tasks like Sudoku, ARC-style proble...

This episode explores a 2024 paper on the LPU, a custom processor designed specifically for large language model inference, with an emphasis on reducing the per-token delay that users notice in interactive systems. It explains why autoregressive decoding is often limited by memory movement and synchronization rather than raw compute, making conventional GPU strengths less decisive in small-batch,...

Episode title: After Titans: Behrouz on Nested Learning and Hope The followup to our Titans episode, by the same core team a year later. Behrouz, Razaviyayn, Zhong, and Mirrokni (Google Research) generalize the Titans bet — that long-term memory should be a learnable module updated at test time — into a broader paradigm they call Nested Learning, where a "deep" architecture is really a hierarchy o...

This episode explores the Titans paper’s proposal to pair standard attention with a separate learned long-term memory that updates during inference, aiming to preserve distant information without paying full quadratic attention costs across very long sequences. It situates that idea against earlier approaches such as Neural Turing Machines, Transformer-XL, Compressive Transformers, Memorizing Tran...

This episode explores how a line of recent systems papers culminates in NanoFlow, a serving approach that breaks LLM inference into very small “nano-batches” so different GPU-intensive operations can overlap instead of running in strict sequence. It explains the shift from thinking only about memory bottlenecks such as KV-cache movement and fragmentation toward a more nuanced claim: even if some k...

Episode title: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention A two-paper deep dive framed around the Dwarkesh Patel x Reiner Pope blackboard lecture on training and serving frontier LLMs. The hosts work through "Unified Scaling Laws for Routed Language Models" (Clark et al., DeepMind 2022, arXiv 2202.01169) for the mixture-of-experts side and the DeepSeek sparse-...

This episode explores MegaScale-Infer, a systems paper on serving large mixture-of-experts language models by separating the attention path from the expert feed-forward path and scheduling them independently. It explains why MoE models can look efficient on paper yet still waste GPU capacity in practice, especially during decode, where KV-cache-heavy attention and uneven expert routing create very...

This episode explores the paper’s claim that decoding cost in large language models is driven less by raw parameter counts and more by hardware-level behavior during autoregressive generation, especially memory bandwidth pressure from the KV cache. It explains why metrics like total or activated parameters can be misleading cost proxies, and walks through the tradeoffs among standard attention, gr...

This episode explores a systems paper on speeding up Transformer decoding by tightly fusing the SwiGLU MLP path, rather than focusing only on attention or long-context tricks. It explains why long output generation becomes memory-bandwidth bound, clarifying concepts like kernel fusion, HBM traffic, prefill versus autoregressive decode, and why repeated token-by-token inference exposes the MLP as a...

This episode explores how the FlashFuser paper uses Hopper GPU inter-core communication to push kernel fusion beyond the usual single-SM memory limits, especially for transformer feed-forward networks and gated FFNs. It explains why this matters now: H100-class GPUs have gained compute far faster than memory bandwidth, making activation spills to HBM an increasingly painful bottleneck for workload...

This episode explores a systems paper on making reinforcement-learning post-training for large language models practical over ordinary Ethernet and even WAN links, rather than requiring expensive RDMA clusters. It explains why trainer-actor RL creates a synchronization bottleneck, how full policy refreshes can dominate runtime on 1 to 10 gigabit networks, and why that turns bandwidth into a hidden...

This episode explores JANUS, a systems approach to serving mixture-of-experts transformers efficiently by separating attention layers from expert layers instead of deploying the whole model as a single monolithic unit. It explains why MoE models can still be expensive and latency-prone in practice: even if only a few experts activate per token, the system must still manage large expert memory foot...

This episode explores whether code models should allocate training data evenly across programming languages or tune the mix based on how each language scales. It walks through the paper’s core experiments on monolingual scaling, bilingual transfer, translation-style code pairing, and multilingual token allocation, with close attention to languages like Python, JavaScript, Rust, and TypeScript. The...

This episode explores Causal-JEPA, a world-modeling approach that masks whole object trajectories rather than image patches to force a model to reason about interactions between entities. It explains how the method combines object-centric representations with JEPA-style latent prediction, asking the model to reconstruct hidden objects from scene context and then predict future dynamics, instead of...

This episode explores Ministral 3, a family of 3B, 8B, and 14B long-context multimodal models built from a 24B parent through structured pruning and cascade distillation rather than separate full-scale training runs. It explains how the method works step by step, from teacher-student distillation and capacity-gap concerns to the staged pruning pipeline that extends each child model to 256k context...

This episode explores a May 13, 2026 arXiv paper arguing that many-shot chain-of-thought prompting can act less like simple retrieval and more like a form of test-time learning, where structured context helps a model reason during inference without changing its weights. It examines the paper’s main claims that adding many reasoning demonstrations does not reliably help across all settings, that se...

This episode explores a position paper arguing that agentic AI systems, built from task decomposition, routing, specialized components, and explicit graph-like workflows, may offer a more credible path to AGI than simply scaling a single monolithic model. It examines how the paper frames AGI through both broad competence across environments and efficient skill acquisition, then asks whether real-w...

This episode explores a 2026 paper on defending language models against unauthorized distillation by rewriting chain-of-thought traces before they are returned through an API. It explains the core idea of making reasoning outputs remain useful and correct for human users while becoming less effective as training data for a smaller model trying to copy the teacher, and it connects that strategy to...

Episode title: Air Force One, Jensen Huang, and Anthropic's 2028 Memo A close reading of Anthropic's policy post "2028: Two Scenarios for Global AI Leadership" as what it actually is — a carefully timed corporate advocacy document, not a peer-reviewed paper. The episode unpacks the post's central "distillation attacks" framing, distinguishes the four very different things that label gets used for,...

This episode explores TMAS, a framework for scaling test-time reasoning by coordinating multiple specialized agents instead of simply letting a single model think longer. It explains how the system combines proposal, verification, refinement, and shared hierarchical memory so that useful intermediate results and higher-level strategy guidance can be reused across parallel reasoning attempts. The d...