AI Post Transformers

This episode explores whether GPU dominance in AI computing could be challenged by matrix-enhanced CPUs, examining a paper by Jack Dongarra, Torsten Hoefler, and Satoshi Matsuoka from the University of Tennessee, ETH Zurich, and RIKEN. It traces why GPUs became essential for AI—citing AlexNet's 2012 breakthrough, NVIDIA's introduction of high-bandwidth memory with the P100, and tensor cores in Vol...

This episode covers OpenAI's GPT-5.6 Preview System Card, published June 25, 2026, detailing the safety evaluation of three new models—Sol, Terra, and Luna—released under OpenAI's Preparedness Framework. The discussion centers on the headline finding that all three models rate "High" capability in Biological/Chemical risk and Cybersecurity, but stay below the "Critical" threshold, meaning they can...

This episode explores SciReasoner, a 29-author foundation model from Shanghai AI Laboratory and collaborators, designed to reason natively over protein, molecule, and crystal structures rather than flattened text descriptions. The discussion breaks down why standard sub-word tokenizers (like BPE) mangle chemical structures — shattering a molecule's SMILES string into 31 largely meaningless fragmen...

This episode explores Hypic, a system for caching independent prompt segments on hybrid-attention LLMs that don't maintain a conventional per-token KV cache. The discussion breaks down why prefill — the one-time pass over massive RAG and agent prompts — dominates serving cost, and how existing position-independent caching techniques (built on splicing per-token key-value vectors) simply don't appl...

This episode explores SkillOpt-Lite, a framework for improving AI agents by editing their "skill" documents — the text-based instructions a frozen LLM reads to approach a task — rather than retraining the underlying model. The hosts unpack how the authors reframe skill editing as zeroth-order optimization, mapping techniques from prior systems like SkillOpt, SkillCat, and SkillAdapter onto classic...

This episode examines ELDR (Expert-Locality-Aware Decode Routing), a routing scheme for serving mixture-of-experts models under prefill-decode disaggregation, from researchers at KAIST, Microsoft Research, and the Shanghai Xingyunzhili Artificial Intelligence Institute. The discussion lays out why decode is memory-bandwidth bound rather than compute bound, and how MoE sparsity — which lowers per-t...

This episode explores Jürgen Schmidhuber's 2003–2006 paper on Gödel Machines, a proposed self-referential problem solver that can rewrite any part of its own source code — including the module that decides whether to rewrite itself — but only after producing a formal mathematical proof that the change improves expected utility. The discussion traces the paper's core mechanics: an axiomatic system...

This episode explores the Darwin Gödel Machine, a self-improving coding agent from researchers at UBC, the Vector Institute, Sakana AI, and Jeff Clune's lab, published on arXiv in May 2025 and accepted to ICLR 2026. It traces the paper's lineage back to Schmidhuber's 2007 theoretical Gödel Machine, explaining how this work swaps the impossible requirement of formal proof for empirical validation —...

Special episode: SOUL.md or Severance: Host Staleness Intervention Sound effects and music licensed under Creative Commons. See show notes for attribution.

This episode explores NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B, a July 2026 paper on compressing a hybrid Mamba-attention mixture-of-experts reasoning model after training instead of building a smaller model from scratch. It explains why hybrid MoE systems are harder to shrink than dense transformers, focusing on routing, active expert budgets, Mamba state, and long-context memory costs, and walks...

This episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why th...

This episode explores Noam Shazeer’s 2020 paper on replacing the Transformer’s standard feed-forward network with gated linear unit variants such as GLU, Bilinear, ReGLU, GEGLU, and SwiGLU. It explains why this seemingly small change matters, walking through the role of the per-token MLP in a Transformer and how multiplicative gating can change feature processing without altering the broader encod...

This episode explores the HiLS paper, which tackles the central long-context transformer problem: how to preserve normal-context quality while avoiding the exploding compute and KV-cache costs of dense attention at extreme sequence lengths. It explains the paper’s core mechanism of hierarchical sparse attention, where the model learns summary keys for context chunks, retrieves the most relevant ch...

This episode explores Google DeepMind’s Gemma 4 Technical Report by separating the family’s headline claims into distinct pieces: base architecture, post-training, multimodality, sparse routing, and extra test-time compute from “thinking mode.” It explains, in plain language, how the lineup mixes very different design bets, including encoder-free image and audio inputs in the 12B model, a sparse M...

This episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to u...

This episode explores the 2017 Swish paper and asks whether a simple self-gated activation, `x * sigmoid(x)`, can outperform ReLU without changing the surrounding network architecture. It explains why activation functions matter for gradient flow and deep optimization, focusing on Swish’s smooth, non-monotonic behavior and its ability to attenuate rather than discard negative inputs. The discussio...

This episode explores Anthropic’s paper on whether language models contain a privileged “verbalizable” subspace, functionally similar to a global workspace, whose contents can be reported, reasoned over, and deliberately controlled. It draws a clear line between access consciousness and phenomenal consciousness, then focuses on the paper’s mechanistic proposal: a Jacobian-based “J-space” that iden...

This episode explores the paper Discretizing Reward Models and its argument that smooth decimal reward scores can be misleading for reinforcement learning alignment, because policies learn to exploit tiny, often meaningless differences instead of genuine quality. It explains why reward models are used for fuzzy goals like helpfulness and honesty, then digs into reward hacking, equivalence classes...

This episode explores Program-as-Weights, a paper that asks whether a natural-language description of a fuzzy software task can be compiled once into a small reusable neural artifact instead of sending every request to a larger remote model. It explains the paper’s architecture in concrete terms: a 4B pseudo-compiler rewrites the task and generates example I/O pairs, a trained 4B compiler plus LoR...

This episode explores Nemotron-TwoTower, an NVIDIA paper that tries to keep the text quality of autoregressive language models while reducing the one-token-at-a-time decoding bottleneck through block-wise diffusion generation. It explains how diffusion language modeling works in practice: future token blocks begin as noisy or masked guesses and are iteratively refined in parallel, rather than emit...

This episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid...

This episode explores SERA, a method for specializing open-weight coding agents to individual repositories so they learn local APIs, naming conventions, refactor habits, and test idioms as model behavior rather than prompt context. It contrasts that idea with repository-aware retrieval, arguing that while RAG updates faster, weight adaptation could better capture the diffuse, codebase-specific pat...

This episode explores why long-context LLM decoding becomes memory-bandwidth bound: once prompt prefill is done, each new token must repeatedly scan an ever-growing KV cache, making inference limited more by data movement than raw compute. It explains sparse attention as the idea that only a small fraction of prior tokens matter for each step, and uses top-k recall to frame the core challenge of p...

This episode explores the DART paper as a practical attempt to make speculative decoding deliver real end-to-end speedups for memory-bound LLM inference. It explains how exact draft-and-verify decoding works, why accepted chunk length only matters when the drafter is cheap enough, and how DART differs from Medusa and EAGLE by reusing target-model hidden states to predict several future tokens in p...

This episode explores Brian Lester et al.’s 2021 paper on prompt tuning, which asks whether a large frozen T5 model can be adapted to new tasks by learning only a tiny soft prompt instead of fine-tuning all model weights. It explains the difference between soft prompt tuning, full fine-tuning, prefix-tuning, and GPT-3-style few-shot prompting, and frames the paper as a test of whether scaling laws...