AI Post Transformers

This episode explores Shampoo, a 2018 optimization algorithm from Google Brain and Princeton that brings second-order curvature information to neural network training without the prohibitive cost of a full preconditioner. The discussion traces the lineage from Newton's method through AdaGrad's diagonal and full-matrix variants, explaining how Shampoo exploits the natural tensor structure of neural...

This episode explores the paper "Scalable Second Order Optimization for Deep Learning" and its introduction of Distributed Shampoo, a Kronecker-factored second-order optimizer that cuts training steps in half compared to a well-tuned Adam baseline on WMT'14 English-to-French translation, with further gains shown on BERT, Criteo click-through-rate modeling, and ResNet-50. The discussion traces the...

This episode explores a paper proposing SCALENET, a hypernetwork-based fix for unsupervised test-time adaptation (TTA) in large language models. It examines why naive per-prompt gradient updates are unstable — a 70-billion-parameter Llama model's negative log-likelihood balloons from 2.21 to 11.49 after just five adaptation steps — and traces the problem to high-variance single-sample gradients th...

This episode explores TwinQuant, a 4-bit post-training quantization method for large language models that challenges a core assumption behind prior techniques like SVDQuant: that a weight matrix's important information can be captured in a small, fixed set of directions. The hosts explain how LLM weight outliers turn out to be spread across hundreds of directions rather than concentrated, forcing...

This episode explores Kohei Honda's tutorial and survey "Model Predictive Control via Probabilistic Inference," which unifies two decades of scattered research—path integral control, reinforcement learning theory, and variational inference—into a single coherent framework called PI-MPC. The discussion traces why classical gradient- and Hessian-based MPC solvers break down on contact-rich robotics,...

This episode explores catastrophic forgetting and plasticity loss in RL-trained language models, and introduces "Fast-Slow Training," a method combining slow weight updates (RLVR) with fast in-context learning to address both. The hosts unpack the distinction between RLVR's automatic, verifiable rewards and traditional RLHF, then dig into two separate failure modes of pure RL post-training: models...

This episode explores a survey on continual learning in large language models, examining how models can be updated after pretraining without the prohibitive cost of full retraining or the risk of catastrophic forgetting — the phenomenon where new training quietly degrades performance on tasks a model previously handled well. The discussion breaks down the problem across three distinct LLM training...

This episode explores a paper examining what happens to continual learning problems when LLM agents shift from parametric updates to memory-augmented architectures. Rather than accepting the industry assumption that external memory sidesteps catastrophic forgetting entirely, the researchers run classic continual-learning protocols on memory-based agents and find the same core problem resurfaces in...

This episode explores a new test-time training method called In-Place TTT, which repurposes the down-projection matrix inside a model's existing gated MLP as adaptable "fast weights," letting a pretrained model keep learning during inference without any architectural changes. A key innovation is replacing the reconstruction-style training target used in prior TTT approaches with an LM-aligned targ...

This episode explores Model-Agnostic Meta-Learning (MAML), the 2017 approach from Chelsea Finn, Pieter Abbeel, and Sergey Levine that trains a single, architecture-agnostic initialization capable of fast adaptation across image classification, regression, and reinforcement learning. Rather than learning a task-specific update rule like earlier recurrent meta-learners, MAML optimizes the starting w...

This episode revisits Teuvo Kohonen's 1972 paper "Correlation Matrix Memories," which reframes associative memory as a hardware fault-tolerance problem rather than a representation-learning one. Kohonen builds a memory from outer-product sums of key and data vectors, then shows mathematically how much recall quality degrades when connections are randomly dropped (an "incomplete" correlation matrix...

This episode explores TFGN, an architectural approach to continual pre-training of large language models that claims to solve catastrophic forgetting without four common crutches: replay buffers, task identifiers, small-scale toy benchmarks, and external penalty terms like Fisher-information regularization. The hosts trace the lineage of the forgetting problem back to 1989, explain why popular fix...

This episode explores StrataCL, a fabric-native communication library from researchers at Peking University, ICT-CAS, UCAS, Shanghai Jiao Tong University, and Huawei, tested on Huawei's CloudMatrix384 supernode. The discussion centers on how communication overhead — which the paper puts at 30-45% of end-to-end time in distributed LLM training and up to 50% at scale — can be cut by giving collectiv...

This episode explores SnapStream, a technique from SambaNova Systems for compressing KV caches during long-sequence LLM decoding on dataflow accelerators, demonstrated at production scale with a 671-billion-parameter DeepSeek-R1 deployment running 128K-token context at over 1,800 tokens per second. The discussion covers why established training-free KV cache eviction methods like SnapKV and Stream...

This episode explores cross-instance attention in disaggregated LLM serving, focusing on the surprising size inversion created by Multi-head Latent Attention: a routed decoding query shrinks to roughly a kilobyte while the cache chunk it must read can balloon to 61 megabytes across layers, upending the old assumption that query and cache are comparably sized. The discussion traces why this scenari...

This episode explores a challenge to conventional wisdom in parameter-efficient fine-tuning, examining a method called MiCA that inverts the logic behind LoRA (Low-Rank Adaptation). Rather than letting trainable weight-update matrices drift freely, as standard LoRA does, MiCA deliberately anchors one matrix to the minor singular-value directions of a weight matrix — the low-energy, rarely-used "co...

This episode examines "Silicon Showdown," a study comparing Nvidia discrete-GPU and Apple unified-memory architectures for running large language models on consumer hardware, tested across model sizes from 1.5 billion to 80 billion parameters. It explains why Nvidia's VRAM Wall forces a stark trade-off between quantizing models down or offloading to slower system RAM across a PCIe bottleneck, whil...

This episode covers "DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch," which tackles a hidden cost in dynamic sparse KV-cache systems: the GPU-resident bookkeeping state (landmarks, reconstructed keys) used to make host-memory offloading fast can itself consume up to 64% of GPU memory — 8.5 times larger than the actual sparse KV entries it's meant to retrieve. Drawing on...

This episode surveys how large language model serving systems manage the key-value cache — the memory storing every token's key and value vectors — as it has grown from a disposable per-request tensor into a resource actively managed, moved, and contended for across GPUs, nodes, and storage tiers. Drawing on a Texas Tech University paper classifying over thirty existing systems, the hosts unpack t...

This episode explores FreeAct, a new approach to quantizing large language models down to 4-bit weights and activations (W4A4), presented by researchers from the National University of Singapore, Huawei Technology, and Central South University. The discussion traces how prior methods like QuaRot and FlatQuant rely on a rigid one-to-one pairing between a rotation matrix applied to activations and i...

This episode explores adaptive verification for speculative decoding when the target model is a sparse Mixture-of-Experts (MoE) system rather than a dense transformer, focusing on the paper "Making Every Verified Token Count." The discussion traces the lineage from Leviathan et al.'s original speculative decoding through tree-based drafting methods like Medusa and EAGLE-3, then explains why MoE ar...

Sitting down with a two-author-plus-one chemical-engineering-rooted textbook on Model Predictive Control, this episode unpacks why the field treats real-time feasibility as a hard constraint rather than a nice-to-have — walking through how MPC re-solves an optimization problem from scratch every control cycle using a known dynamics model, with no learning or reward signal involved. The discussion...

This episode explores DWDP (Distributed Weight Data Parallelism), a new NVIDIA-authored approach to LLM inference on NVL72 systems that targets a subtle but costly inefficiency: GPUs sitting idle while they wait to synchronize with slower peers. The hosts unpack how existing model-parallelism strategies—expert, tensor, and pipeline parallelism—all share a hidden flaw, forcing every GPU to hit a sy...

This episode explores "Lifelong Learning of Large Language Model based Agents: A Roadmap," a survey examining how AI agents can continuously adapt to changing environments without losing prior knowledge. The discussion centers on the stability-plasticity dilemma—the tension between preserving learned capabilities and remaining flexible enough to absorb new information—and how this classical proble...

This episode explores why open-weight LLMs like Llama 3.1, Gemma3, Qwen3, and Olmo3 systematically lose 11–39% relative accuracy on facts from 2023–2024 compared to facts from 2020–2021, even though the more recent data falls within their training window. Drawing on Kyutai's paper "Understanding Data Temporality Impact on Large Language Models Pre-training," the discussion traces this "knowledge h...