AI Post Transformers

This episode explores a paper that argues AI can help mathematics most by orchestrating the full research workflow rather than acting as a one-shot chatbot. It discusses why real mathematical work depends on durable memory, branching hypotheses, literature search, proof attempts, computation, and recorded failures, and contrasts that with both ordinary chat interfaces and formal theorem provers su...

This episode explores MiA-Signature, a long-context reasoning method that argues models should approximate a broad, query-driven activation pattern over memory rather than rely on narrow top-k retrieval. It explains how the paper builds a two-stage pipeline: first retrieving a wide pool of potentially relevant context, then compressing that pool into a compact signature of high-level concepts chos...

This episode explores a May 7, 2026 arXiv paper on Lighthouse Attention and asks whether long-context language models can be pretrained cheaply with hierarchical sparse attention, then switched back to standard dense attention late in training without losing dense-model quality. It explains why long-context training is so expensive even with FlashAttention, contrasting dense quadratic attention wi...

This episode explores the paper δ-mem, which argues that long context windows are not the same as true memory and proposes a compact online memory module for frozen language models. It explains how the method uses a tiny mutable state matrix, updated with a delta rule, to store residual errors over time and feed that state back into generation as a low-rank attention correction rather than replayi...

This episode explores the Qwen-Image-2.0 technical report and its central claim that a single unified multimodal diffusion model can handle high-quality image generation, precise image editing, multilingual text rendering, ultra-long text, and complex instruction following in one system. It traces the technical background from vision transformers and latent diffusion through newer editing and text...

This episode explores MELT, a looped language model architecture that aims to preserve latent reasoning benefits while preventing KV-cache memory from growing with every reasoning pass. It explains how the paper reframes the problem as a systems and architecture challenge, replacing per-loop cached attention state with a single shared, gated cache per layer inspired by recurrent models like LSTMs...

This episode explores a 2026 paper arguing that the real trust problem in language models is not error alone, but confident error, and that improving trust may depend more on metacognition than on simply scaling up knowledge. It unpacks key distinctions such as knowledge boundaries, calibration, discrimination, and the gap between intrinsic uncertainty and the uncertainty a model expresses in word...

This episode explores TIDE, a transformer variant that lets every layer re-access the original token identity instead of relying entirely on contextual hidden states to preserve that information. It explains how this design targets rare-token failures and “contextual collapse,” where tokens appearing in similar contexts can become too hard for the model to distinguish, especially in scientific, bi...

This episode explores a paper on test-time scaling that asks whether an LLM agent can automatically discover better inference-time control policies than the hand-built heuristics researchers usually rely on. It explains the core search framework in concrete terms: a controller decides when to branch, continue, probe, prune, or stop, balancing reasoning depth, breadth, and compute budget rather tha...

This episode explores ELF: Embedded Language Flows, a continuous-time diffusion language model that stays in embedding space until the final decoding step instead of repeatedly snapping back to discrete tokens during generation. It explains how that design lets the model borrow flow-matching and guidance techniques from image diffusion, while arguing that earlier continuous text models may have un...

This episode explores ForkKV, a systems paper on serving multiple LoRA-based agents from one base language model without duplicating massive KV caches for shared context. It explains why ordinary prefix caching breaks once different LoRA adapters change the activations, then walks through the paper’s core idea: split cache state into a large shared base and a small adapter-specific residual, using...

This episode explores whether CXL memory expansion has finally become practical for hyperscale production, using the 2025 Vistara system as a case study. It explains the core ideas behind CXL, tiered memory, and memory disaggregation, then argues that the real comparison is not against swap but against transparent page placement that keeps hot data in local DRAM and colder pages in a slower expand...

This episode explores a paper arguing that a model’s internal answer belief can form well before its visible chain-of-thought reveals it, raising doubts about whether reasoning traces are true explanations or polished post hoc narratives. It explains core ideas such as chain-of-thought faithfulness, activation monitoring, mechanistic interpretability, confidence calibration, and dynamic inference,...

This episode explores RAPTOR, a method for extracting concept directions from language model hidden states using ridge-regularized logistic probes, with the goal of making those directions accurate enough for interpretation and stable enough for activation steering. It explains the core probe-then-steer workflow, why linear probes can reveal what a model has encoded, and why good classification ac...

This episode explores a 2026 paper on “split personality training,” a method for attaching an internal reviewer to a language model that can reveal what the model knows about its own deceptive or reward-hacking behavior without changing the answer shown to the user. It situates the work in the broader lineage of latent knowledge elicitation, alignment faking, and mechanistic interpretability, expl...

This episode explores a mechanistic interpretability paper arguing that transformers often fail at counting not because they lack an internal notion of quantity, but because the pathway that converts that latent count into digit tokens is poorly aligned. It explains key ideas like linear probes, the logit lens, attention, LoRA, constrained next-token evaluation, and autoregressive generation to sh...

This episode explores a historical argument that simple Hebbian learning rules are too weak to explain real intelligence, because they mainly capture local correlations rather than solving multivariate credit-assignment problems. It examines how that critique points toward global optimization methods such as backpropagation and, in some settings, reinforcement learning, while contrasting their eng...

This episode explores LAPS, a serving system for large language models that treats long prompt prefills and short multi-turn re-prefills as fundamentally different workloads instead of batching them together. It explains why user-perceived latency, especially time to first token, suffers when tiny follow-up requests get stuck behind large compute-heavy context loads, and how LAPS models the bounda...

This episode explores a 2026 paper, Generative Modeling via Drifting, which argues that the hard transport process behind modern generative models can be moved into training so that inference becomes a single forward pass. It explains the core idea of a pushforward distribution, introduces the paper’s notion of a drifting field that nudges generated samples toward the data distribution during opti...

This episode explores a March 2026 paper arguing that LLM-based judges are an unreliable way to measure jailbreak success and adversarial robustness. It explains how modern safety evaluations rely on judge models to score harmful outputs, then walks through why those judges can break under attack shift, model shift, and data shift, sometimes degrading to near coin-flip reliability. The discussion...

This episode explores EverMemOS, a memory system for long-lived AI agents that tries to organize past interactions into structured, higher-level semantic “scenes” instead of relying on flat retrieval alone. It explains why bigger context windows and standard RAG often fail when agents accumulate stale preferences, conflicting facts, and fragmented conversational traces, arguing that the real probl...

This episode explores a position paper arguing that modern LLM serving has outgrown simple heuristics like FIFO, shortest-queue routing, and LRU eviction. It explains why transformer inference creates harder control problems than standard inference, focusing on continuous batching, KV-cache growth, and the tension between compute-heavy prefill and memory-bound decode phases. The discussion highlig...

This episode explores a paper on long-context compression that argues standard “soft compression” methods, which rely on learned memory or gist tokens, lose information because those tokens get overwritten across layers and fail to coordinate what each slot should retain. It explains the paper’s alternative design, which keeps the language model backbone frozen and instead explicitly transmits inf...

This episode explores CL-BENCH, a benchmark designed to test whether language models can actually learn task-specific knowledge from long, messy context and then reason with it, rather than merely retrieving facts or mimicking examples. It explains the distinction between long-context understanding, in-context learning, and the stronger notion of context learning, using examples like legal codes,...

This episode explores SGLang, a system for making complex language model workflows run faster by treating them as full programs rather than single prompt-response calls. It explains how modern LLM applications involve branching, tool use, retries, and structured outputs, then examines SGLang’s co-design of a Python-embedded language with a specialized runtime that can optimize those patterns direc...