AI Post Transformers

This episode explores Atlas, a 2025 paper on test-time memorization that asks whether a model with fixed recurrent memory can learn to update that memory during inference and rival Transformers on long-context recall and reasoning. It explains the core tradeoff between Transformer-style KV caches, which preserve near-exact token access at growing cost, and bounded recurrent memory, which must deci...

This episode explores KumoRFM, a 2025 proposal for a foundation model that can perform in-context learning directly on relational databases, aiming to handle tasks like churn prediction, fraud detection, recommendation, and forecasting without training a separate model for each schema and label. It explains how the approach represents warehouse data as heterogeneous graphs of rows and foreign-key...

This episode explores Latent Reasoning with Normalizing Flows, a paper that asks whether a standard left-to-right transformer can do its intermediate reasoning in continuous latent states instead of spelling every step out as text. It explains how the method uses a frozen VAE during training to compress written rationales into short latent sequences, then uses shallow normalizing flows so the same...

This episode explores the RAGEN-2 paper’s claim that agentic reinforcement learning can produce reasoning traces that look active and diverse while losing real dependence on the input. It explains the paper’s central distinction between ordinary entropy, which measures diversity within a single prompt, and template collapse, where traces across many different prompts become generic variations of t...

This episode explores Unified Neural Scaling Laws, a framework for predicting model performance when parameter count, data volume, training steps, inference compute, and training-recipe choices all change at once. It explains how the paper moves beyond classic smooth power-law curves by introducing broken multivariate scaling laws with regime shifts, including hyperbreaks, bottleneck versus non-bo...

This episode explores KumoRFM-2, a relational foundation model designed to learn directly from connected database tables instead of flattening customers, orders, products, and tickets into a single feature table. It explains why relational learning matters for enterprise tasks such as churn, fraud, and demand prediction, arguing that flattening often erases multi-hop relationships, repeated intera...

This episode explores Predictive Query Language (PQL), a SQL-shaped domain-specific language for defining supervised learning tasks directly over relational databases by specifying the prediction target, entity, and future time horizon in one declarative statement. It explains why training label generation is often the real bottleneck in applied machine learning, unpacking concepts like prediction...

This episode explores why decoder-style language models can generate fluent text yet still underperform dedicated embedding models when asked for zero-shot sentence vectors, despite embeddings being critical infrastructure for search, retrieval-augmented generation, clustering, and recommendation. It examines the paper’s main argument that the problem is not just bad pooling or prompting, but a de...

This episode explores End-to-End Context Compression at Scale, a paper on whether learned context compression can beat the cost of long-context inference in quality, time to first token, and peak memory. It explains the main design choices behind the authors’ Latent Context Language Models, which use a 0.6B encoder and 4B decoder to replace long token sequences with learned latent memory at compre...

This episode explores EMO, a Mixture-of-Experts language model that tries to turn token-level sparsity into real modularity by letting coherent expert groups emerge from document structure during pretraining. It explains why standard MoEs are not automatically deployable as smaller task-specific slices, then walks through EMO’s main idea: each document routes tokens through a learned document-spec...

This episode explores Harvest, a system for LLM inference that uses idle HBM on neighboring NVLink-connected GPUs as a temporary cache when a serving GPU runs out of local memory. It explains why LLM serving is often bottlenecked more by memory capacity and data movement than by raw compute, focusing on two concrete cases: growing KV caches during long-context decoding and the shifting expert weig...

This episode explores SFMP, a post-training, weight-only quantization method for large language models that targets awkward memory budgets like 3.25 or 3.75 bits per weight, where standard 3-bit or 4-bit schemes are a poor fit. It explains the core ideas behind mixed-precision quantization, including salience scoring with diagonal Fisher information, block-wise allocation, and why hardware constra...

This episode explores VeriCache, a systems paper that asks whether a large language model can draft tokens using a compressed, lossy KV cache and then verify them against the full cache to recover exactly the same greedy-decoding output. It explains why KV cache size has become a central bottleneck in long-context inference, unpacking token dropping, KV quantization, prefix caching, and the specul...

This episode explores DeepMind’s paper on technical AGI safety and security, focusing on how labs might prevent severe, humanity-scale harm before highly capable systems are deployed. It breaks down the paper’s core distinctions between misuse and misalignment, explains what the authors mean by Exceptional AGI and the no-human-ceiling assumption, and examines dangerous capability evaluations in ar...

This episode explores Mooncake, a production LLM serving architecture that treats KV cache reuse and movement as the central challenge in long-context chat, not just raw GPU compute. It explains why prefill and decode stress hardware in different ways, how metrics like time to first token and time between tokens drive system design, and why separating those phases helps meet real latency targets....

This episode explores Marina Favaro et al.’s 2026 paper on whether AI is beginning to accelerate frontier AI development enough to hint at recursive self-improvement, laying out the ladder from chatbots and coding agents to systems that materially help build their own successors. It distinguishes progress on long-horizon, fixed-goal engineering tasks from the harder problem of genuine research jud...

This episode explores a 2026 paper on whether language models can keep doing long, step-by-step reasoning internally or eventually need to expose some of that reasoning in visible chain-of-thought tokens. It explains the paper’s core idea of opaque serial depth, or a model’s hidden reasoning horizon, and argues that this is a better safety-relevant measure than raw model size, parameter count, or...

This episode explores why batch-1 LLM decode for robots, edge copilots, and other single-session agents behaves very differently from high-throughput serving, and why next-token latency cannot be explained by memory bandwidth alone. It breaks down the paper’s main test: compare real decode time against an analytic memory floor based on model-weight and KV-cache traffic, then run that across Qwen-2...

This episode explores TRELLIS, a bounded-memory transformer architecture that replaces the usual ever-growing key-value cache with a fixed set of learned memory slots that are rewritten during inference. It explains why long-context serving is constrained less by training-time quadratic attention than by the linear growth, latency, and fragility of KV caches, and situates TRELLIS in the progressio...

This episode explores a post-training method for making mixture-of-experts language models cheaper at inference time without retraining them from scratch. It explains how the paper converts a fully trained static MoE into a dynamic one by adding parameter-free zero experts, allowing some tokens to skip normal experts, and then uses self-distillation to preserve the original model’s behavior under...

This episode explores SmolLM2, a 1.7 billion parameter language model from Hugging Face that tries to compete with stronger small models not by changing the transformer architecture, but by radically improving the training data mix and sequencing across roughly 11 trillion tokens. It explains the distinction between pretraining and instruction tuning, then argues that for compact models, dataset q...

This episode explores Google’s Snap system, which moves major host-networking functions out of the kernel and into isolated userspace services while trying to keep the performance benefits usually associated with kernel bypass. It examines why that shift mattered operationally at fleet scale: kernel networking changes could take one to two months to deploy, while Snap enabled roughly weekly releas...

This episode explores a paper proposing that language models could handle long-context reasoning by periodically pausing, replaying soon-to-be-evicted context offline, and consolidating it into fixed-size fast-weight memory instead of carrying an ever-growing KV cache. It explains the core machinery behind the idea, including state space models and Gated Delta Networks, and clarifies why this is m...

This episode explores the 2008 Dragonfly network topology paper and why its ideas suddenly matter again for large-scale AI systems in 2026. It explains how Dragonfly uses high-radix routers and router groups to keep most traffic to a local hop, a single global hop, and another local hop, reducing the number of expensive long-distance optical links compared with flattened butterfly and folded Clos...

This episode explores KVzap, a method for pruning transformer KV caches by learning a cheap surrogate for a much stronger oracle, with the goal of making cache eviction practical during both prompt prefilling and token-by-token decoding. It explains why KV caches dominate long-context inference costs, clarifies the difference between prefilling and decoding, and lays out why serving systems have f...