AI Post Transformers

This episode explores "Context Language Models," a paper that lets a model rewrite its own context window with ordinary Bash or Python edits, instead of relying on harness-defined truncation or a fixed menu of context actions. It covers the three tiers of context control, the serving catch (an edit in the middle or at the start of the context can cost several times more prefill compute than an app...

This episode examines LongAttnComp, a method that uses a small trained scorer to pick about 16,000 relevant tokens out of roughly 115,000. The goal is for other companies' models to find code bugs about as well as with the full input. It explains why long prompts are costly (prefill dominates time-to-first-token) and how extractive compression differs from summarization. It also ties the approach...

This episode examines AttnComp, a method that uses a language model's own middle-layer attention, lightly fine-tuned, to compress retrieved documents in retrieval-augmented generation. It covers why large retrieved contexts hurt: noisy passages can push accuracy below no retrieval, key facts get lost in the middle, and every extra token adds cost. It contrasts slow abstractive compressors with ext...

This episode examines "Jenga," a paper from the team behind vLLM's PagedAttention, which asks what that memory manager costs on modern models that break its two core assumptions: that every layer stores identically sized per-token state, and that every layer needs the full prefix. It covers sliding-window layers, Mamba-style recurrent states, and cross-attention vision models. In a Llama 3.2 Visio...

This episode examines ATLAS, a simulator for 3D-DRAM-based LLM accelerators that the authors say was checked against real silicon. It explains why decoding is memory-bound on GPUs, and how hybrid bonding shrinks the data path from millimeters to micrometers. That cuts access energy by 77 to 83 percent compared with HBM. It covers how the design differs from processing-in-memory, and the trade-offs...

This episode examines an ISCA 2026 paper on early silicon for a logic-on-3D-DRAM inference card, which targets about 100 TB/s of memory bandwidth for generative decoding. It explains why decode is memory-bound, using a Llama-70B KV-cache example that grows to roughly 20 GB per user at 128K context. It then compares three memory options: SRAM (fast but tiny), HBM (large but bandwidth-starved), and...

This episode examines a policy paper asking what happens if AI automates AI research itself, setting off an "intelligence explosion" that compresses years of progress into months. It reviews the evidence that automation is already underway, including the AI share of approved code at one lab rising from low single digits to over 80 percent, autonomous R&D work growing from 1 to 26 percent, and task...

This episode examines "Large Memory Layers with Product Keys," which replaces a transformer's feed-forward network with a sparse memory of over a million trainable slots. The layer's central trick is splitting each query in half and scoring it against two small codebooks of sub-keys. This avoids scoring every key and cuts search cost by roughly a thousandfold. The hosts work through a toy example...

This episode examines Chakra, an MLCommons effort to define one open, graph-based execution-trace format. The format is meant to carry a distributed AI workload from production observation through benchmark replay to hardware simulation, without exposing a proprietary model. It explains how a trace is a per-device directed acyclic graph of compute, memory, and communication nodes linked by control...

This episode examines "Memory Layers at Scale" from Meta FAIR, which asks whether a language model can gain huge numbers of fact-storing parameters without adding any arithmetic per token. It explains how a memory layer works: a token's query retrieves the top-k rows from a large learned key-value table, and product keys keep the search cheap. It also covers where the idea came from, including the...

This episode examines the Coconut paper, which trains a language model to feed its last hidden state back in as the next input, so it reasons in continuous "thoughts" instead of written words. It first covers why chain-of-thought forces every step through a single discrete token, then sets Coconut against nearby approaches. Those are pause tokens, filler tokens, iCoT's gradual removal of reasoning...

This episode examines "Thinking Without Words," a paper proposing Abstract Chain-of-Thought, which replaces long written-out reasoning with a short run of up to 128 tokens drawn from a new 64-token codebook that has no meaning in English. It sets the approach against its predecessors: pause tokens, which tend to fail when added after pre-training, stepwise internalization, and continuous latent me...

This episode explores Model-Aware Tokenizer Transfer, a method for swapping a language model's tokenizer without breaking it. The motivating example is Gemma 3 on Ukrainian, where a Ukrainian-focused vocabulary raises compression from about 2.98 to 4.44 characters per token, a near 50% inference speedup. The cost is more than 80,000 new tokens with no trained embeddings. The discussion explains wh...

This episode examines RedFuser, an Alibaba Cloud framework that automatically fuses cascaded reductions, such as softmax feeding a matrix multiply, into a single GPU kernel. It starts with why chained reductions resist standard compiler fusion: each stage needs the finished result of the one before, which forces repeated memory reads and prevents overlap. The discussion then works through the onli...

This episode examines the vLLM Semantic Router paper, which treats request routing as a layer of named "signals" (keywords, language, domain, complexity, modality, PII, jailbreak, fact-checking need) combined into Boolean decision policies that pick a model and run plugins such as semantic caching or hallucination checks. It compares this with earlier routers like RouteLLM, RouterDC and AutoMix, w...

This episode explores a vision paper arguing that choosing a model, routing a request, and sizing a GPU fleet are a single coupled problem. The paper organizes LLM inference into workload, router, and pool dimensions, and it presents a 3-by-3 matrix of their interactions. The discussion defines the latency and serving terms (TTFT, TPOT, prefill versus decode, KV-cache limits, warm versus cold requ...

This episode examines a March 2026 paper on the vLLM Semantic Router. The paper claims to cut routing time for an 8K-token prompt from about 4.9 seconds to 50 milliseconds without a dedicated GPU. It explains how the router runs jailbreak, PII and domain classifiers inside Envoy before the first token. It also explains why those classifiers must fit in a few hundred megabytes of GPU memory left ov...

This episode examines "Alternating Updates for Efficient Transformers" (Baykal et al., Google Research and UC Berkeley, NeurIPS 2023), a technique that reports up to an 87% inference speedup on SQuAD at matched accuracy without changing attention mechanisms or reducing parameter count. The discussion contrasts this approach, called AltUp, with Mixture-of-Experts methods like Switch Transformer and...

This episode examines a theory paper on manifold-constrained hyper-connections (mHC), which widen the residual stream into four parallel streams and force each layer's mixing matrix to be doubly stochastic using Sinkhorn normalization. It explains why the composite of many mixers looks nearly uniform, and why that suggests the extra width fades with depth. The key argument is that the average of t...

This episode examines LayerRoPE, a 2026 preprint proposing that the steady growth of hidden-state norms with depth in Transformers is a learned depth-positional code rather than a defect to suppress. It lays out the background: residual-stream accumulation under Pre-Norm, massive activations, high-norm tokens in ViTs, and the Curse of Depth. It also covers remedies such as Layer-Norm Scaling, Peri...

This episode examines "HBF Sucks?", a simulation study of what happens when High-Bandwidth Flash (stacked 3D NAND with a wide package-local interface) replaces the SSD tier in a Mooncake-style KV-offloading stack, and why serving gets slower. It first covers the KV cache, prefix reuse, and the near-tier/far-tier split, then HBF's tradeoffs: flash-scale capacity and aggregate bandwidth, but microse...

This episode examines a September 2026 paper that asks whether the roughly 33.5% jump in arXiv Mathematics list entries (35,307 to 47,127, January through August) reflects a real generative-AI effect or just a growing platform. It walks through a "ladder" of estimates. Raw growth is 33.5%, growth over Mathematics' own pre-2022 trend is 36.8%, and growth over a synthetic-control counterfactual buil...

This episode dives into Agensh, a Microsoft Research paper that scales leaderless AI coding-agent organizations up to 1,024 concurrent workers on the same codebase. The discussion frames why every current multi-agent coding framework—Claude Code's sub-agents, Codex, Copilot's fleet command, Kimi's Agent Swarm—hits the same wall: a single orchestrator agent that plans, assigns, and integrates work...

This episode examines SPLASH, a design from the National University of Singapore that co-designs sparse attention with High-Bandwidth Flash (HBF), NAND dies stacked next to the GPU, to serve million-token contexts. The starting problem is KV cache capacity: 64 concurrent million-token requests to Qwen3-235B need about 12.9 TB, enough to fill the HBM of 68 B200 GPUs. The discussion shows why flash...

This episode examines "Disaggregated Quantization: Specializing LLM Prefill and Decode," which argues that prefill (compute-bound) and decode (memory-bound) should each get their own quantization format, weights, and storage placement instead of sharing one compromise. The discussion covers why weight-only formats like GPTQ suit decode while W4A4 formats like QuaRot and NVFP4 suit prefill, and why...