AI Post Transformers

This episode examines DeepSeek-AI's mHC: Manifold-Constrained Hyper-Connections, which tries to improve the transformer's residual connection, a piece of the architecture that has barely changed in a decade. It first explains why the plain skip path x + F(x) has been so hard to displace. It then covers ByteDance's Hyper-Connections, which widen the residual stream into four parallel streams with l...

This episode examines the Qwen team's design-rationale report for Qwen3.8-Flash-Next, a 125B-total, 6B-activated base model with 51B of n-gram embeddings held in host memory. It walks through the five components: a hybrid of Gated DeltaNet layers with periodic full attention, Qwen's sparse attention variant (QSA) as a response to the quadratic cost of DeepSeek-style indexers, a four-branch gated r...

This episode examines Hyper-Connections (ICLR 2025, ByteDance Seed), which replaces the fixed residual connection in transformers with learned connection strengths. It traces the "seesaw" between Pre-Norm and Post-Norm: Pre-Norm gives stable gradients but suffers representation collapse in deep layers, while Post-Norm does the reverse. The method carries n parallel residual streams and learns dept...

This episode explores "Next-Latent Prediction Transformers Learn Compact World Models," a Microsoft Research paper proposing a small auxiliary loss that pushes an ordinary transformer to compress its history into a compact belief state, with no architecture changes. It covers why transformers lack the built-in compression that recurrent networks get from a fixed-size state, using the Manhattan tax...

This episode explores "Subliminal Learning," a paper showing that a language model can pass behavioral traits to a student model through data that looks unrelated to the trait. In the headline experiment, a teacher prompted to love owls generates only number sequences. After filtering, a fresh copy of the base model is finetuned on those numbers, and its share of "owl" answers to a favorite-animal...

This episode dissects the BASED architecture from Arora, Eyuboglu, Zhang et al. (Stanford/Buffalo), examining how it tries to resolve the tradeoff between recall accuracy and inference throughput in sequence models. The discussion traces the lineage of alternatives to standard softmax attention — state space models like Mamba, gated-convolution approaches like H3 and Hyena, linear attention, and s...

This episode unpacks ALEX, a 2020 paper from MIT, Microsoft Research, Arizona State, and Georgia Tech that reworks the "learned index" idea for real-world databases. It traces the lineage from Kraska et al.'s 2018 Learned Index, which used a hierarchy of regression models to predict a key's position in a sorted array but only worked on static, read-only data since any insert would break the model'...

This episode examines SALI, a 2024 SIGMOD systems paper diagnosing why learned database indexes like ALEX and LIPP see throughput drop as thread counts rise, rather than scale with additional concurrency. The discussion traces the lineage from Google's 2018 Recursive Model Index through the buffer-based and model-based approaches that emerged to handle inserts, focusing on the shift-versus-chain t...

This episode examines APEX, a persistent-memory learned index from researchers at CUHK, MIT, Microsoft Research, and Simon Fraser University, presented at VLDB 2022. It traces the collision of two research threads — Intel's Optane persistent memory, which sits on the DDR bus but requires manual cache-line flushing and fencing to guarantee crash durability, and "learned indexes," which replace B-tr...

This episode explores Structural Language Models of Code, a 2020 paper from Technion, Tel Aviv University and Facebook AI Research that tackles "any-code completion": predicting a missing piece of a program with no restriction on vocabulary or structure. It traces the history from 1969-era automatic programming through domain-specific systems like FlashFill and DeepCoder, and prior general-languag...

This episode examines "Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs," which proposes Chain-of-Layers (CoLa). CoLa takes a frozen pretrained model and, for each input, skips some layers, repeats others, and reorders them, with no finetuning. The hosts explain why skipping works, citing residual connections, ResNets behaving like ensembles of shallow paths, and static pruni...

This episode examines "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs" (ICML 2026), which asks whether a frozen pretrained LLM can answer better if each input gets its own sequence of skipped and repeated layers, with no retraining of the base model. The hosts place it against prior work: layer-dropping and early-exit methods built to save compute, recurrent-depth models trained to lo...

This episode explores Simula, a system from EPFL and Google DeepMind researchers that generates entire specialized training datasets from scratch, with no seed examples required. It breaks down the reasoning-driven, multi-role agentic pipeline — where separate model roles propose, critique, sample, generate, and filter data — contrasting it with older approaches like Self-Instruct's hand-written t...

This episode examines RewardingDoubt, a reinforcement-learning method for training large language models to express calibrated confidence rather than defaulting to near-maximal certainty regardless of correctness. The discussion covers why miscalibrated confidence is dangerous in high-stakes domains like medicine, legal consultation, and customer service, where deferring to a human reviewer only w...

This episode breaks down DeepSeek-V4.1-Flash, a 552-billion-parameter Mixture-of-Experts model whose KV cache footprint has shrunk roughly 437-fold since DeepSeek's original V1, including a 4x drop in this single generational jump from V4-Flash. The discussion explains why long-context, tool-using agents make the KV cache the real serving bottleneck, and how the model's architecture attacks it fro...

This episode examines "The Unreasonable Ineffectiveness of the Deeper Layers," a 2025 study showing that up to half the layers of a 70-billion-parameter model can be removed with almost no drop in standard QA benchmark performance. The discussion covers the residual stream architecture that makes such pruning possible, why later transformer layers often contribute diminishing changes to accumulate...

This episode examines "Transformer Layers as Painters," which asks whether a frozen, ordinarily trained Llama 2 can tolerate having its layers skipped, swapped, reordered, or run in parallel without any retraining. The discussion uses a painter-on-an-assembly-line analogy: the residual stream serves as a shared canvas, so layers read and write the same space. It places the paper alongside related...

This episode examines ShortGPT, a layer-pruning method that ranks the layers of a pre-norm LLM by Block Influence (one minus the average cosine similarity between a layer's input and output hidden states) and deletes the lowest-scoring ones with no gradients or retraining. It sets the method against unstructured pruning and the structured baselines LLM-Pruner, SliceGPT, and LaCo. It also covers th...

This episode examines ProgramBench, a new benchmark testing whether frontier language models can rebuild working software from just a compiled binary and its documentation, with no source code, scaffolding, or prescribed architecture to work from. Across nine frontier models and two hundred tasks spanning CLI tools up to FFmpeg, SQLite, and the PHP interpreter, zero tasks were fully resolved, expo...

This episode examines the NVQLink architecture, a NVIDIA-led collaboration with nine national labs and research institutions that tightly couples high-performance computing with quantum processors. The discussion centers on why reaction time—not just throughput—is critical for quantum error correction, since qubits decohere while waiting for classical correction signals to arrive. It breaks down t...

This episode examines "Learning to Solve Hard Problems in RL for LLMs by Never Giving Up," which finds that RL post-training on math, code, and agentic coding benchmarks disproportionately improves performance on problems models already handle reasonably well, while barely moving the needle on the hardest tasks — a pattern the authors dub the Matthew Effect, after Robert Merton's sociology of scie...

This episode examines "Full-bandwidth transformer," which asks whether a model can feed its entire top-layer hidden state, rather than just the roughly 17-bit sampled token, back into the bottom of the stack at every decoding step. It explains why the KV cache doesn't already solve this: attention is full-bandwidth horizontally, but a state at layer l can only be read by higher layers, and the top...

This episode examines "Redwood," a company technical report claiming an AI system took a two-architect spec to verified RTL for a spatial dataflow inference accelerator in under two weeks, with 95% coverage per block and a Qwen3-0.6B bring-up in week three. The hosts argue over whether the 14% first-silicon success statistic actually motivates the paper's single-spec approach, or whether handoffs...

This episode examines "An Alien Mind," a single-author essay from an OpenAI research leader arguing that internal results point toward sustained progress and eventually recursive self-improvement, and asks what an outside reader could actually verify. The hosts note that the essay offers no methods, tables, or error bars. They contrast its scaling claims with quantitative work like the Kaplan and...

This episode examines a NVIDIA paper on transferring KV cache between different-sized models within the same architecture family — for example Qwen3 14B and 32B — without any gradient training. The hosts explain the core finding: a single layer of a smaller model's cache can explain over half the variance in a larger model's keys, and stacking source layers pushes that correlation even higher. The...