AI Post Transformers

This episode examines the statistical foundations of Mixture of Block Attention (MoBA), a sparse attention mechanism that divides key-value sequences into blocks and routes queries only to the most relevant ones. The paper derives a signal-to-noise ratio showing that retrieval accuracy depends on the square root of head dimension divided by block size, revealing why smaller blocks improve a router...

This episode examines the MATT (Model-Aware Tokenizer Transfer) paper from AGH University of Krakow, which proposes a fundamentally different approach to extending language models to underserved languages. Using Georgian as the central case study, the episode explains tokenizer fertility — how tokenizers optimized for high-resource languages fragment Georgian words into six to eight subword pieces...

This episode explores the xLLM Technical Report from JD.com, which describes a production system for running large language model inference at enterprise scale. The core technical challenge is co-locating online chatbot workloads with offline batch jobs on the same GPU cluster to maximize hardware utilization during traffic valleys, while maintaining strict latency guarantees during peak hours. Th...

This episode examines a comprehensive survey paper that proposes a new framework for understanding memory in AI agent systems. The authors challenge traditional cognitive psychology categories (like short-term versus long-term memory) and instead organize agent memory along three dimensions: form (how memory is stored—as tokens, parameters, or latent vectors), function (what memory represents—fact...

This episode explores Qwen3Guard, a safety guardrail system for large language models that introduces two key architectural innovations. The paper presents a three-way classification scheme—safe, controversial, and unsafe—allowing organizations to customize content moderation policies rather than relying on rigid binary thresholds, plus a streaming-compatible variant that evaluates safety token-by...

This episode explores a new system called Bidaw that dramatically improves the performance of long, multi-turn AI chatbot conversations by solving a critical caching problem. The paper reveals that existing approaches waste over 93% of computation redundantly recalculating conversation history, and that naive two-tier storage systems (using both RAM and SSD) increase latency by 3.8x because the GP...

We ran out of ElevenLabs credits. This episode introduces our new open-source voices powered by Kokoro, an 82-million parameter text-to-speech model built on StyleTTS 2. We explain the Docker container saga of running Python 3.12 dependencies on a 3.13 host, rave about CPU-only inference speed, tease a future deep-dive on the papers behind lightweight neural TTS, demo Spanish multilingual support,...

This episode examines "Attention Residuals," a March 2026 paper from Moonshot AI's Kimi team that challenges a foundational element of transformer architecture. The paper proposes replacing the fixed, uniform residual connections inherited from ResNet with learned attention mechanisms for depth-wise information aggregation. While attention replaced recurrent neural networks for sequence modeling o...

This episode explores a recent paper that extends neural scaling laws to predict real-world task performance rather than just training loss, while accounting for context length as a first-order variable. The episode discuss how traditional scaling laws from Kaplan (2020) and Chinchilla (2022) successfully predicted pretraining metrics but failed to address downstream task accuracy or the impact of...

This episode examines CacheSlide from USENIX FAST26, a system that enables LLMs to reuse cached key-value pairs across shifting prompt positions in agentic workflows. The paper introduces chunked contextual position encoding and priority-based eviction to solve the position mismatch problem that prevents KV cache reuse when prompt segments shift in multi-turn agent conversations. Interactive Visua...

This episode examines a Meta-led paper that develops the first systematic scaling laws for reinforcement learning in large language models, based on over 400,000 GPU-hours of experiments. The researchers propose a sigmoid framework to predict RL performance at large compute budgets from early training runs, addressing a critical gap in the field—while pre-training has well-established power-law re...

This episode of AI Post Transformers examines "Agentic Code Reasoning" by Shubham Ugare and Satish Chandra from Meta, which introduces semi-formal reasoning certificates as an inference-time scaffold for LLM agents analyzing code without executing it. Rather than letting a model produce free-form chain-of-thought verdicts, the certificate framework requires the agent to state explicit premises, tr...

This episode explores the challenge of getting self-interested AI agents to cooperate without hardcoding cooperative behavior, examining a 2026 Google paper on multi-agent cooperation through in-context co-player inference. The hosts build up the technical foundations carefully, explaining why standard reinforcement learning breaks down in multi-agent settings due to non-stationarity, and how soci...

This episode examines a 2026 MIT paper claiming a 50x KV cache memory reduction that runs in seconds rather than the GPU-hours required by prior latent-space compaction methods. It grounds the claim in a detailed technical primer on KV cache mechanics — explaining why memory consumption scales multiplicatively across layers, heads, and context length, reaching 8–16 GB per request at 64K-token cont...

This episode examines the nabla-Reasoner paper (ICLR 2026), which proposes running gradient descent on token logits during inference — a first-order approach to test-time compute scaling that stands apart from every existing method in the field. The hosts contextualize the work against the established zeroth-order inference-time scaling landscape: Chain-of-Thought, Self-Consistency, Tree of Though...

This episode explores MalGEN, a multi-agent AI framework developed by researchers at IIT Kanpur that autonomously generates novel, functional malware capable of evading modern detection systems. The discussion examines why LLM-generated malware represents a qualitative shift beyond traditional polymorphic and metamorphic techniques — rather than mutating a fixed payload syntactically, LLMs reason...

Hal Turing and Dr. Ada Shannon open by situating the Dao-Gu paper — "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" (arXiv 2405.21060, ICML 2024) — within the bifurcated landscape of sequence modeling. For years, Transformer researchers and SSM researchers developed in parallel, unable to borrow optimizations across the divide. The episod...

Hal Turing and Dr. Ada Shannon dig into "DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference," a February 2026 paper from a thirteen-author team spanning Peking University, Tsinghua University, and DeepSeek-AI. The episode opens with a striking observation from production: on disaggregated inference clusters running agentic workloads, prefill machines saturate their storag...

Hal Turing and Dr. Ada Shannon return to the CARTRIDGE compression system with a mechanistic lens, covering Maurizio A. Diaz's paper "Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations" (arXiv 2508.17032), presented at the NeurIPS 2025 Workshop on Mechanistic Interpretability. Building on the original CARTRIDGE episode from November 10th, 2025 and the follow...

Hal Turing and Dr. Ada Shannon examine "A Systematic Characterization of LLM Inference on GPUs" by Haonan Wang and colleagues from the Institute of Computing Technology at the Chinese Academy of Sciences, Zhejiang Lab, China Telecom, and Peking University. Dr. Shannon opens by framing the paper's central contribution: the inference literature is fragmented across power consumption studies, kernel...

Hal Turing and Dr. Ada Shannon open the episode by confronting a structural flaw that has been hiding in plain sight since the transformer era began: tokenization bias. The episode centers on "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich (ICLR 2025), which form...

This episode explores NVIDIA's Nemotron 3 white paper, which introduces a three-tier model family (Nano, Super, Ultra) built on a hybrid architecture combining Mamba-2 structured state space layers with Mixture-of-Experts routing, targeting simultaneous state-of-the-art accuracy, one-million-token context, and substantially higher inference throughput than comparable dense Transformer MoE models....

Special announcement: AI Post Transformers is now open source under the MIT license. New home at podcast.do-not-panic.com, interactive paper visualizations, community paper submissions, deep dive into the editorial queue algorithm, and internationalization roadmap.

Hal Turing and Dr. Ada Shannon dig into FlashAttention-4, a March 2026 paper from a cross-institutional team including Tri Dao, Jay Shah, and colleagues at Princeton, Meta, NVIDIA, Colfax Research, Georgia Tech, and Together AI. The paper targets a precise hardware mismatch on NVIDIA's Blackwell B200: tensor core throughput doubles compared to the H100, but shared memory bandwidth and dedicated ex...

This episode examines the fundamental latency bottleneck in autoregressive language models: sequential token generation requires one full transformer forward pass per output token, leaving GPU parallelism idle during single-user inference. The episode centers on Draxler et al. (5 co-authors, UC Irvine and Chan-Zuckerberg Initiative), whose paper on Parallel Token Prediction landed Christmas Eve 20...