AI Post Transformers

This episode explores "Subliminal Learning," a paper showing that a language model can pass behavioral traits to a student model through data that looks unrelated to the trait. In the headline experiment, a teacher prompted to love owls generates only number sequences. After filtering, a fresh copy of the base model is finetuned on those numbers, and its share of "owl" answers to a favorite-animal...

This episode dissects the BASED architecture from Arora, Eyuboglu, Zhang et al. (Stanford/Buffalo), examining how it tries to resolve the tradeoff between recall accuracy and inference throughput in sequence models. The discussion traces the lineage of alternatives to standard softmax attention — state space models like Mamba, gated-convolution approaches like H3 and Hyena, linear attention, and s...

This episode unpacks ALEX, a 2020 paper from MIT, Microsoft Research, Arizona State, and Georgia Tech that reworks the "learned index" idea for real-world databases. It traces the lineage from Kraska et al.'s 2018 Learned Index, which used a hierarchy of regression models to predict a key's position in a sorted array but only worked on static, read-only data since any insert would break the model'...

This episode examines SALI, a 2024 SIGMOD systems paper diagnosing why learned database indexes like ALEX and LIPP see throughput drop as thread counts rise, rather than scale with additional concurrency. The discussion traces the lineage from Google's 2018 Recursive Model Index through the buffer-based and model-based approaches that emerged to handle inserts, focusing on the shift-versus-chain t...

This episode examines APEX, a persistent-memory learned index from researchers at CUHK, MIT, Microsoft Research, and Simon Fraser University, presented at VLDB 2022. It traces the collision of two research threads — Intel's Optane persistent memory, which sits on the DDR bus but requires manual cache-line flushing and fencing to guarantee crash durability, and "learned indexes," which replace B-tr...

This episode explores Structural Language Models of Code, a 2020 paper from Technion, Tel Aviv University and Facebook AI Research that tackles "any-code completion": predicting a missing piece of a program with no restriction on vocabulary or structure. It traces the history from 1969-era automatic programming through domain-specific systems like FlashFill and DeepCoder, and prior general-languag...

This episode examines "Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs," which proposes Chain-of-Layers (CoLa). CoLa takes a frozen pretrained model and, for each input, skips some layers, repeats others, and reorders them, with no finetuning. The hosts explain why skipping works, citing residual connections, ResNets behaving like ensembles of shallow paths, and static pruni...

This episode examines "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs" (ICML 2026), which asks whether a frozen pretrained LLM can answer better if each input gets its own sequence of skipped and repeated layers, with no retraining of the base model. The hosts place it against prior work: layer-dropping and early-exit methods built to save compute, recurrent-depth models trained to lo...

This episode explores Simula, a system from EPFL and Google DeepMind researchers that generates entire specialized training datasets from scratch, with no seed examples required. It breaks down the reasoning-driven, multi-role agentic pipeline — where separate model roles propose, critique, sample, generate, and filter data — contrasting it with older approaches like Self-Instruct's hand-written t...

This episode examines RewardingDoubt, a reinforcement-learning method for training large language models to express calibrated confidence rather than defaulting to near-maximal certainty regardless of correctness. The discussion covers why miscalibrated confidence is dangerous in high-stakes domains like medicine, legal consultation, and customer service, where deferring to a human reviewer only w...

This episode breaks down DeepSeek-V4.1-Flash, a 552-billion-parameter Mixture-of-Experts model whose KV cache footprint has shrunk roughly 437-fold since DeepSeek's original V1, including a 4x drop in this single generational jump from V4-Flash. The discussion explains why long-context, tool-using agents make the KV cache the real serving bottleneck, and how the model's architecture attacks it fro...

This episode examines "The Unreasonable Ineffectiveness of the Deeper Layers," a 2025 study showing that up to half the layers of a 70-billion-parameter model can be removed with almost no drop in standard QA benchmark performance. The discussion covers the residual stream architecture that makes such pruning possible, why later transformer layers often contribute diminishing changes to accumulate...

This episode examines "Transformer Layers as Painters," which asks whether a frozen, ordinarily trained Llama 2 can tolerate having its layers skipped, swapped, reordered, or run in parallel without any retraining. The discussion uses a painter-on-an-assembly-line analogy: the residual stream serves as a shared canvas, so layers read and write the same space. It places the paper alongside related...

This episode examines ShortGPT, a layer-pruning method that ranks the layers of a pre-norm LLM by Block Influence (one minus the average cosine similarity between a layer's input and output hidden states) and deletes the lowest-scoring ones with no gradients or retraining. It sets the method against unstructured pruning and the structured baselines LLM-Pruner, SliceGPT, and LaCo. It also covers th...

This episode examines ProgramBench, a new benchmark testing whether frontier language models can rebuild working software from just a compiled binary and its documentation, with no source code, scaffolding, or prescribed architecture to work from. Across nine frontier models and two hundred tasks spanning CLI tools up to FFmpeg, SQLite, and the PHP interpreter, zero tasks were fully resolved, expo...

This episode examines the NVQLink architecture, a NVIDIA-led collaboration with nine national labs and research institutions that tightly couples high-performance computing with quantum processors. The discussion centers on why reaction time—not just throughput—is critical for quantum error correction, since qubits decohere while waiting for classical correction signals to arrive. It breaks down t...

This episode examines "Learning to Solve Hard Problems in RL for LLMs by Never Giving Up," which finds that RL post-training on math, code, and agentic coding benchmarks disproportionately improves performance on problems models already handle reasonably well, while barely moving the needle on the hardest tasks — a pattern the authors dub the Matthew Effect, after Robert Merton's sociology of scie...

This episode examines "Full-bandwidth transformer," which asks whether a model can feed its entire top-layer hidden state, rather than just the roughly 17-bit sampled token, back into the bottom of the stack at every decoding step. It explains why the KV cache doesn't already solve this: attention is full-bandwidth horizontally, but a state at layer l can only be read by higher layers, and the top...

This episode examines "Redwood," a company technical report claiming an AI system took a two-architect spec to verified RTL for a spatial dataflow inference accelerator in under two weeks, with 95% coverage per block and a Qwen3-0.6B bring-up in week three. The hosts argue over whether the 14% first-silicon success statistic actually motivates the paper's single-spec approach, or whether handoffs...

This episode examines "An Alien Mind," a single-author essay from an OpenAI research leader arguing that internal results point toward sustained progress and eventually recursive self-improvement, and asks what an outside reader could actually verify. The hosts note that the essay offers no methods, tables, or error bars. They contrast its scaling claims with quantitative work like the Kaplan and...

This episode examines a NVIDIA paper on transferring KV cache between different-sized models within the same architecture family — for example Qwen3 14B and 32B — without any gradient training. The hosts explain the core finding: a single layer of a smaller model's cache can explain over half the variance in a larger model's keys, and stacking source layers pushes that correlation even higher. The...

This episode examines CacheBridge, a paper proposing targeted fixes to a training-free method for transferring KV caches between different transformer models in multi-model routing setups. It explains why caches can't simply be handed off — differing residual widths, GQA head counts, and RoPE position encoding make one model's cache unreadable to another — and how a prior affine-mapper approach (F...

This episode explores a Google DeepMind paper proposing that separately trained language models can exchange raw internal state — the key-value cache built during transformer inference — through a shared "global latent space," rather than communicating only through text. The hosts unpack why text is a lossy bottleneck for inter-model communication, and how lightweight, frozen-weight adapter pairs...

This episode examines "Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models" by Jin-woo Lee and six co-authors from Chungnam National University and KISTI, which tackles the problem of transferring one model's KV cache — its layer-by-layer key/value memory of a processed document — to a completely different model architecture without re-running the original text...

This episode examines "Dream-RSI: Recursive Self-Improvement through Evolving Worlds," which proposes making the exploration strategy of a discovery system — not the underlying model — the target of recursive self-improvement. Rather than optimizing candidate solutions directly, the system optimizes the policy that decides how to search: which branches to expand, how to parallelize workers, and wh...