AI Post Transformers

Hal Turing and Dr. Ada Shannon take a deep dive into EMO: Pretraining Mixture of Experts for Emergent Modularity, a May 7, 2026 paper by Ryan Wang and co-authors from UC Berkeley and the Allen Institute for AI. The episode centers on a practical deployment question: if a workload is mostly code, math, or biomed, why must operators keep an entire giant model in memory instead of loading only the re...

This episode explores a 2026 paper on stabilizing deep reinforcement learning by pushing an agent’s hidden representations toward an isotropic Gaussian shape. It explains how nonstationarity in RL, from shifting data distributions, bootstrapped targets, and primacy bias, can make agents overfit early experience, lose plasticity, and accumulate dormant neurons. The discussion focuses on the paper’s...

This episode explores Social Simulacra, a method for using large language models to prototype entire online communities before they exist by generating synthetic members, posts, and reply threads from a community goal, rules, and a small set of seed personas. It explains why that matters for social computing: small pilots can miss emergent failures like norm drift, newcomer enculturation problems,...

This episode explores a 2025 study on fine-tuning large language models to predict how people respond in social science experiments, asking whether trained models can simulate new studies more reliably than prompting alone. It explains how the researchers built SOCSCI210, a dataset of 2.9 million responses from more than 400,000 participants across 210 TESS experiments, and why standardizing those...

This episode explores the UIST 2025 paper "Creating General User Models from Computer Use," which proposes building a persistent user model from raw computer traces such as screenshots, UI text, message context, and app switching. It explains how the system stores confidence-weighted natural-language propositions about a person’s preferences, knowledge, goals, and current situation, aiming to supp...

This episode explores a paper on building reusable LLM-based simulations of specific individuals by grounding agents in people’s own interviews, survey responses, or both, rather than relying on thin demographic personas. It explains how the system was tested on 1,052 Americans using holdout evaluations across survey questions, personality traits, behavioral experiments, and randomized interventio...

This episode explores Weak-SIGReg, a lightweight covariance regularizer designed to prevent representation collapse in fragile supervised training, especially for small-data Vision Transformers. It explains how the method uses a sketched covariance matrix and an identity-matching penalty to keep hidden features decorrelated and similarly scaled at much lower cost than full covariance regularizatio...

Hal Turing and Dr. Ada Shannon examine an empirical study of parameter-efficient fine-tuning for large language models, centered on a practical question: when does a small task-specific update beat retraining the entire model? Using FLAN-T5-XL as the test bed, they frame PEFT as a transfer-learning strategy that freezes most of the transformer while learning a compact adaptation layer, whether thr...

This episode explores OpenSkill, a framework for LLM agents that tries to improve behavior after deployment by building durable, reusable skills from public evidence rather than retraining model weights. It explains how the paper separates ordinary tool use from open-world self-evolution, arguing that the key challenge is not just acting with browsers and code, but turning documentation, repositor...

This episode explores NVIDIA’s Nemotron 3 Ultra, an open 550-billion-parameter mixture-of-experts model with only 55 billion parameters active per token, a hybrid Mamba-transformer backbone, and a 1 million token context window aimed at long-running agentic reasoning. It explains how sparse MoE routing, Mamba-style sequence layers, and low-precision NVFP4 training are used to cut KV-cache pressure...

This episode explores SageAttention2, an ICML 2025 paper on making exact transformer attention faster without changing the underlying computation, focusing on why long-context models still pay a steep quadratic cost and why exact kernels remain important despite sparse and linear alternatives. It explains the paper’s central claim that aggressive low-precision attention can work only with careful...

This episode explores Inclusion AI’s Ling and Ring 2.6 technical report, which asks how a trillion-parameter model can stay fast, handle very long contexts, and remain dependable in multi-step agent workflows. It explains why agentic AI makes latency and token costs much more painful than in ordinary chat, especially when models must carry long instruction traces, tool outputs, and large working c...

This episode explores how quantization affects reasoning models, asking how much weights, activations, and KV caches can be compressed before multi-step reasoning starts to fail. It explains the main quantization strategies in practical serving terms, from weight-only methods like AWQ and GPTQ to weight-activation schemes such as W8A8 and W4A4, and KV cache compression for long decoding traces. Th...

This episode explores InfiniGen, a systems approach to speeding up long-context language model inference by treating KV cache management, not raw compute, as the central bottleneck. It explains why decoding slows down when large caches have to shuttle between CPU and GPU memory, and contrasts InfiniGen with FlexGen, H2O, and PagedAttention to show how different serving setups create different memo...

This episode explores DafnyBench, a benchmark for testing whether large language models can help with one of formal verification’s hardest practical bottlenecks: reconstructing the missing assertions and loop invariants that make Dafny programs verifiable. It explains how formal verification differs from ordinary testing and from theorem proving, and why the paper deliberately frames the task as r...

This episode explores the 2009 seL4 paper and why formally verifying an operating-system kernel matters when that kernel sits at the center of every higher-level security claim. It explains how seL4’s microkernel design keeps only core mechanisms like threads, IPC, interrupts, and memory objects in privileged code, contrasting that with monolithic kernels and showing why minimality makes theorem-p...

This episode explores DafnyPro, a system for using large language models to help verify Dafny programs while keeping the original executable logic unchanged. It explains the basics of formal verification in Dafny, including preconditions, postconditions, loop invariants, decreases clauses, ghost code, and why writing correct proof annotations is much harder than generating plausible code. The disc...

This episode explores the paper Cartridges at Scale, which asks whether large document collections can be distilled into reusable modular KV-cache memories so a model can answer questions without repeatedly rereading raw text. It explains what a cartridge is, how context distillation turns full-document context into compact learned prefixes, and why that differs from prompt caching, fine-tuning, o...

This episode explores PaperBench, a benchmark designed to test whether frontier AI agents can independently replicate the empirical work of recent machine learning papers from scratch rather than merely explain them. It breaks down what agentic AI actually entails in this setting: reading papers, writing code, choosing baselines, reconstructing missing details, running experiments, debugging failu...

This episode explores MiniMax Sparse Attention, a long-context transformer design that aims to preserve dense-model quality at million-token scale while sharply reducing the quadratic compute and memory costs of standard attention. It explains how the method combines Grouped Query Attention with blockwise sparse retrieval: a lightweight Index Branch scores past context in blocks, forces a recent l...

This episode explores DeepMind’s From AGI to ASI as a foresight report that treats human-level general intelligence not as the endpoint, but as a possible stepping stone toward systems that could outperform entire organizations in planning, research, engineering, and coordination. It breaks down how the paper defines AGI and the much more ambitious idea of ASI, then examines the conceptual tools b...

This episode explores a 2025 paper on using Dafny as a hidden, verification-aware intermediate language for AI code generation, where a model first produces a formal specification and verified implementation before compiling it into ordinary Python. It examines the paper’s central trust claim: formal verification can prove that the generated code satisfies the hidden spec, but it cannot prove that...

This episode explores the ACE proposal from AMD, Intel, and the x86 Ecosystem Advisory Group, which would add matrix-native AI instructions to x86 CPUs so transformer workloads can run dense linear algebra more efficiently without changing the models themselves. It explains why AVX10 and VNNI fall short for GEMM-heavy inference, introducing outer-product updates, 2-D tile registers, and the reuse...

This episode explores a June 2026 paper on when document-specific LoRA adapters actually help compared with standard retrieval-augmented generation, especially once a model’s KV cache has been aggressively compressed. It walks through the core mechanics of RAG, LoRA, prefill vs. decode costs, parametric retrieval augmentation, and the Compactor method used to rank and retain only part of a documen...

This episode explores the paper Test-Time Training with KV Binding Is Secretly Linear Attention and asks whether KV-binding test-time training is really doing online memorization or instead behaving like learned linear attention with sequence-specific fast weights. It explains how this approach differs from a standard transformer KV cache, situates it within earlier test-time-training work on expr...