AI Post Transformers

This episode examines ATLAS, a simulator for 3D-DRAM-based LLM accelerators that the authors say was checked against real silicon. It explains why decoding is memory-bound on GPUs, and how hybrid bonding shrinks the data path from millimeters to micrometers. That cuts access energy by 77 to 83 percent compared with HBM. It covers how the design differs from processing-in-memory, and the trade-offs...

This episode examines an ISCA 2026 paper on early silicon for a logic-on-3D-DRAM inference card, which targets about 100 TB/s of memory bandwidth for generative decoding. It explains why decode is memory-bound, using a Llama-70B KV-cache example that grows to roughly 20 GB per user at 128K context. It then compares three memory options: SRAM (fast but tiny), HBM (large but bandwidth-starved), and...

This episode examines a policy paper asking what happens if AI automates AI research itself, setting off an "intelligence explosion" that compresses years of progress into months. It reviews the evidence that automation is already underway, including the AI share of approved code at one lab rising from low single digits to over 80 percent, autonomous R&D work growing from 1 to 26 percent, and task...

This episode examines "Large Memory Layers with Product Keys," which replaces a transformer's feed-forward network with a sparse memory of over a million trainable slots. The layer's central trick is splitting each query in half and scoring it against two small codebooks of sub-keys. This avoids scoring every key and cuts search cost by roughly a thousandfold. The hosts work through a toy example...

This episode examines Chakra, an MLCommons effort to define one open, graph-based execution-trace format. The format is meant to carry a distributed AI workload from production observation through benchmark replay to hardware simulation, without exposing a proprietary model. It explains how a trace is a per-device directed acyclic graph of compute, memory, and communication nodes linked by control...

This episode examines "Memory Layers at Scale" from Meta FAIR, which asks whether a language model can gain huge numbers of fact-storing parameters without adding any arithmetic per token. It explains how a memory layer works: a token's query retrieves the top-k rows from a large learned key-value table, and product keys keep the search cheap. It also covers where the idea came from, including the...

This episode examines the Coconut paper, which trains a language model to feed its last hidden state back in as the next input, so it reasons in continuous "thoughts" instead of written words. It first covers why chain-of-thought forces every step through a single discrete token, then sets Coconut against nearby approaches. Those are pause tokens, filler tokens, iCoT's gradual removal of reasoning...

This episode examines "Thinking Without Words," a paper proposing Abstract Chain-of-Thought, which replaces long written-out reasoning with a short run of up to 128 tokens drawn from a new 64-token codebook that has no meaning in English. It sets the approach against its predecessors: pause tokens, which tend to fail when added after pre-training, stepwise internalization, and continuous latent me...

This episode explores Model-Aware Tokenizer Transfer, a method for swapping a language model's tokenizer without breaking it. The motivating example is Gemma 3 on Ukrainian, where a Ukrainian-focused vocabulary raises compression from about 2.98 to 4.44 characters per token, a near 50% inference speedup. The cost is more than 80,000 new tokens with no trained embeddings. The discussion explains wh...

This episode examines RedFuser, an Alibaba Cloud framework that automatically fuses cascaded reductions, such as softmax feeding a matrix multiply, into a single GPU kernel. It starts with why chained reductions resist standard compiler fusion: each stage needs the finished result of the one before, which forces repeated memory reads and prevents overlap. The discussion then works through the onli...

This episode examines the vLLM Semantic Router paper, which treats request routing as a layer of named "signals" (keywords, language, domain, complexity, modality, PII, jailbreak, fact-checking need) combined into Boolean decision policies that pick a model and run plugins such as semantic caching or hallucination checks. It compares this with earlier routers like RouteLLM, RouterDC and AutoMix, w...

This episode explores a vision paper arguing that choosing a model, routing a request, and sizing a GPU fleet are a single coupled problem. The paper organizes LLM inference into workload, router, and pool dimensions, and it presents a 3-by-3 matrix of their interactions. The discussion defines the latency and serving terms (TTFT, TPOT, prefill versus decode, KV-cache limits, warm versus cold requ...

This episode examines a March 2026 paper on the vLLM Semantic Router. The paper claims to cut routing time for an 8K-token prompt from about 4.9 seconds to 50 milliseconds without a dedicated GPU. It explains how the router runs jailbreak, PII and domain classifiers inside Envoy before the first token. It also explains why those classifiers must fit in a few hundred megabytes of GPU memory left ov...

This episode examines "Alternating Updates for Efficient Transformers" (Baykal et al., Google Research and UC Berkeley, NeurIPS 2023), a technique that reports up to an 87% inference speedup on SQuAD at matched accuracy without changing attention mechanisms or reducing parameter count. The discussion contrasts this approach, called AltUp, with Mixture-of-Experts methods like Switch Transformer and...

This episode examines a theory paper on manifold-constrained hyper-connections (mHC), which widen the residual stream into four parallel streams and force each layer's mixing matrix to be doubly stochastic using Sinkhorn normalization. It explains why the composite of many mixers looks nearly uniform, and why that suggests the extra width fades with depth. The key argument is that the average of t...

This episode examines LayerRoPE, a 2026 preprint proposing that the steady growth of hidden-state norms with depth in Transformers is a learned depth-positional code rather than a defect to suppress. It lays out the background: residual-stream accumulation under Pre-Norm, massive activations, high-norm tokens in ViTs, and the Curse of Depth. It also covers remedies such as Layer-Norm Scaling, Peri...

This episode examines "HBF Sucks?", a simulation study of what happens when High-Bandwidth Flash (stacked 3D NAND with a wide package-local interface) replaces the SSD tier in a Mooncake-style KV-offloading stack, and why serving gets slower. It first covers the KV cache, prefix reuse, and the near-tier/far-tier split, then HBF's tradeoffs: flash-scale capacity and aggregate bandwidth, but microse...

This episode examines a September 2026 paper that asks whether the roughly 33.5% jump in arXiv Mathematics list entries (35,307 to 47,127, January through August) reflects a real generative-AI effect or just a growing platform. It walks through a "ladder" of estimates. Raw growth is 33.5%, growth over Mathematics' own pre-2022 trend is 36.8%, and growth over a synthetic-control counterfactual buil...

This episode dives into Agensh, a Microsoft Research paper that scales leaderless AI coding-agent organizations up to 1,024 concurrent workers on the same codebase. The discussion frames why every current multi-agent coding framework—Claude Code's sub-agents, Codex, Copilot's fleet command, Kimi's Agent Swarm—hits the same wall: a single orchestrator agent that plans, assigns, and integrates work...

This episode examines SPLASH, a design from the National University of Singapore that co-designs sparse attention with High-Bandwidth Flash (HBF), NAND dies stacked next to the GPU, to serve million-token contexts. The starting problem is KV cache capacity: 64 concurrent million-token requests to Qwen3-235B need about 12.9 TB, enough to fill the HBM of 68 B200 GPUs. The discussion shows why flash...

This episode examines "Disaggregated Quantization: Specializing LLM Prefill and Decode," which argues that prefill (compute-bound) and decode (memory-bound) should each get their own quantization format, weights, and storage placement instead of sharing one compromise. The discussion covers why weight-only formats like GPTQ suit decode while W4A4 formats like QuaRot and NVFP4 suit prefill, and why...

This episode examines a four-page IEEE Computer Architecture Letter proposing that agentic LLM serving split KV cache by temperature: HBM keeps the small "hot" set for sessions actively decoding, while high bandwidth flash (3D NAND stacked behind an HBM-style interface) holds the large "cold" pool of paused sessions. It explains why agent loops of decode, tool call, idle wait and resume leave giga...

This episode examines a May 2026 preprint claiming that chain-of-thought monitoring, where a second model reads a reasoning model's scratchpad to catch reward hacking, breaks down across 13 typologically diverse languages. Sixteen models from 8B to 120B parameters were tested on GPQA science questions with a planted hint pointing to a wrong answer. Deception rates averaged 95.9 percent and reached...

This episode examines "How Far Can Disaggregation Go?", a design-space study of attention–FFN disaggregation (AFD) for serving mixture-of-experts models. It places AFD at the top of a ladder that starts with chunked-prefill aggregation and continues through prefill–decode disaggregation. It explains why attention (memory-bound, KV-cache heavy) and expert FFNs (compute-bound, needing pooled batches...

This episode examines "Scaling Large Language Model-based Multi-Agent Collaboration," which pushes agent-based collaboration to over a thousand coordinated LLM agents, far beyond the handful typically used in prior systems. The discussion traces the paper's intellectual roots in neural scaling laws (Kaplan et al., Chinchilla) and emergent abilities research (Wei et al.), applying that same lens to...