AI Post Transformers

This episode explores NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B, a July 2026 paper on compressing a hybrid Mamba-attention mixture-of-experts reasoning model after training instead of building a smaller model from scratch. It explains why hybrid MoE systems are harder to shrink than dense transformers, focusing on routing, active expert budgets, Mamba state, and long-context memory costs, and walks...

This episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why th...

This episode explores Noam Shazeer’s 2020 paper on replacing the Transformer’s standard feed-forward network with gated linear unit variants such as GLU, Bilinear, ReGLU, GEGLU, and SwiGLU. It explains why this seemingly small change matters, walking through the role of the per-token MLP in a Transformer and how multiplicative gating can change feature processing without altering the broader encod...

This episode explores the HiLS paper, which tackles the central long-context transformer problem: how to preserve normal-context quality while avoiding the exploding compute and KV-cache costs of dense attention at extreme sequence lengths. It explains the paper’s core mechanism of hierarchical sparse attention, where the model learns summary keys for context chunks, retrieves the most relevant ch...

This episode explores Google DeepMind’s Gemma 4 Technical Report by separating the family’s headline claims into distinct pieces: base architecture, post-training, multimodality, sparse routing, and extra test-time compute from “thinking mode.” It explains, in plain language, how the lineup mixes very different design bets, including encoder-free image and audio inputs in the 12B model, a sparse M...

This episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to u...

This episode explores the 2017 Swish paper and asks whether a simple self-gated activation, `x * sigmoid(x)`, can outperform ReLU without changing the surrounding network architecture. It explains why activation functions matter for gradient flow and deep optimization, focusing on Swish’s smooth, non-monotonic behavior and its ability to attenuate rather than discard negative inputs. The discussio...

This episode explores Anthropic’s paper on whether language models contain a privileged “verbalizable” subspace, functionally similar to a global workspace, whose contents can be reported, reasoned over, and deliberately controlled. It draws a clear line between access consciousness and phenomenal consciousness, then focuses on the paper’s mechanistic proposal: a Jacobian-based “J-space” that iden...

This episode explores the paper Discretizing Reward Models and its argument that smooth decimal reward scores can be misleading for reinforcement learning alignment, because policies learn to exploit tiny, often meaningless differences instead of genuine quality. It explains why reward models are used for fuzzy goals like helpfulness and honesty, then digs into reward hacking, equivalence classes...

This episode explores Program-as-Weights, a paper that asks whether a natural-language description of a fuzzy software task can be compiled once into a small reusable neural artifact instead of sending every request to a larger remote model. It explains the paper’s architecture in concrete terms: a 4B pseudo-compiler rewrites the task and generates example I/O pairs, a trained 4B compiler plus LoR...

This episode explores Nemotron-TwoTower, an NVIDIA paper that tries to keep the text quality of autoregressive language models while reducing the one-token-at-a-time decoding bottleneck through block-wise diffusion generation. It explains how diffusion language modeling works in practice: future token blocks begin as noisy or masked guesses and are iteratively refined in parallel, rather than emit...

This episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid...

This episode explores SERA, a method for specializing open-weight coding agents to individual repositories so they learn local APIs, naming conventions, refactor habits, and test idioms as model behavior rather than prompt context. It contrasts that idea with repository-aware retrieval, arguing that while RAG updates faster, weight adaptation could better capture the diffuse, codebase-specific pat...

This episode explores why long-context LLM decoding becomes memory-bandwidth bound: once prompt prefill is done, each new token must repeatedly scan an ever-growing KV cache, making inference limited more by data movement than raw compute. It explains sparse attention as the idea that only a small fraction of prior tokens matter for each step, and uses top-k recall to frame the core challenge of p...

This episode explores the DART paper as a practical attempt to make speculative decoding deliver real end-to-end speedups for memory-bound LLM inference. It explains how exact draft-and-verify decoding works, why accepted chunk length only matters when the drafter is cheap enough, and how DART differs from Medusa and EAGLE by reusing target-model hidden states to predict several future tokens in p...

This episode explores Brian Lester et al.’s 2021 paper on prompt tuning, which asks whether a large frozen T5 model can be adapted to new tasks by learning only a tiny soft prompt instead of fine-tuning all model weights. It explains the difference between soft prompt tuning, full fine-tuning, prefix-tuning, and GPT-3-style few-shot prompting, and frames the paper as a test of whether scaling laws...

This episode explores the paper Is Finer Better? The Limits of Microscaling Formats in Large Language Models and examines why shrinking microscaling block sizes can unexpectedly make low-bit LLM quantization worse instead of better. It walks through how microscaling pairs FP4 weights or activations with shared local FP8 scales, contrasts that setup with coarser quantization schemes, and places the...

This episode explores a 2026 paper on cache-resident LLM inference, asking whether modern CPUs with gigabyte-scale last-level caches can cut decoding latency by keeping model weights on-chip instead of repeatedly fetching them from DRAM. It explains why autoregressive decoding is often memory-bound rather than compute-bound, then breaks down the paper’s main design ideas: separating weight-heavy p...

This episode explores SuperInfer, a system for serving large language models on GH200-style superchips by treating memory management as the key lever for meeting latency targets rather than just maximizing compute use. It explains why KV cache growth, HBM pressure, and head-of-line blocking often hurt responsiveness first, then breaks down how the paper’s RotaSched policy proactively rotates reque...

This episode explores TTKV, a temporal-tiered key-value cache design for long-context LLM inference, where decode speed degrades because growing KV state turns generation into a memory-bandwidth problem rather than a compute problem. It explains how the method keeps recent cache blocks in fast GPU HBM, evicts older blocks to slower host DRAM, and uses asymmetric quantization in the slow tier, pres...

This episode explores Moebius, a serving system for mixture-of-experts transformers that can switch at runtime between tensor parallelism and expert parallelism without restarting or draining live requests. It explains why tensor parallelism tends to give lower latency at low concurrency, while expert parallelism delivers better throughput at high concurrency, making bursty online traffic and RL r...

This episode explores LLMServingSim 2.0, a simulator designed to model how large language models behave when they are served on mixed hardware fleets with separated compute, memory, and networking resources rather than a uniform GPU cluster. It explains the practical serving concepts that shape user experience, including prefill versus decode, time to first token, time per output token, prefix cac...

This episode explores DSpark, a DeepSeek-AI paper on improving speculative decoding by starting from a DFlash-style block-parallel draft model and increasing how often a larger verifier accepts its proposed tokens. It explains the mechanics of speculative decoding in plain language, situates DSpark within earlier blockwise and multi-token prediction work, and notes that the technique is already us...

This episode explores JETSPEC, a 2026 inference paper on speculative decoding that asks whether a language model can draft an entire tree of future tokens in parallel while preserving causal consistency and actually reducing latency on long generations. It explains why autoregressive decoding remains a serving bottleneck for long proofs, code completions, and assistant replies, even when the under...

This episode explores Information-Aware KV Cache Compression for Long Reasoning, a paper about making long-context inference cheaper and more reliable by deciding which KV-cache tokens to keep during extended reasoning. It explains why long prefilling and long decoding turn the cache into a major memory bottleneck, and why common heuristics such as sliding windows or recent-attention-based retenti...