AI Post Transformers

This episode explores how transformers split prediction between knowledge stored in their weights and information inferred from the current prompt, using the paper’s synthetic “bigram world” to make those mechanisms visible. It explains the distinction between global statistical knowledge and true in-context knowledge, then walks through induction heads as a concrete circuit for recalling earlier...

This episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference...

This episode explores a 2026 paper on recursive multi-agent systems that asks whether AI collaboration can scale better by replacing text-based agent communication with shared latent-state updates. It explains the difference between agent topology, assigned roles, and communication channels, and argues that natural language may be a costly bottleneck compared with the richer internal representatio...

This episode explores a review of mechanistic interpretability for transformer language models, focusing on how researchers study internal features, circuits, and claims of universality across models. It explains the core toolkit behind the field, including linear probes, hidden-state analysis, intervention methods, vocabulary projection, and sparse autoencoders, while grounding those ideas in tra...

This episode explores DeltaKV, a method for reducing the huge GPU memory burden of KV caches in long-context language model inference without simply discarding old tokens. It contrasts three strategies for handling long contexts: token eviction, dynamic sparse attention, and true compression, arguing that the cache contains structured redundancy that can be exploited rather than treated as disposa...

This episode explores CacheFlow, a systems approach to speeding up long-context LLM serving by restoring transformer KV caches more intelligently. It explains the tradeoffs between recomputing prior attention state, loading it from storage, or combining both, and argues that the real user-facing bottleneck is now time-to-first-token rather than raw generation speed. The discussion focuses on Cache...

This episode explores a speech recognition paper that replaces slow left-to-right transcript generation with a faster draft-and-edit approach, where a speech model produces an initial hypothesis and a bidirectional LLM corrects it in parallel. It explains the tradeoff between CTC-based systems, which are fast but weaker at using linguistic context, and autoregressive decoders, which are more expre...

This episode explores whether large language models can genuinely recognize the limits of their own knowledge or whether they have simply learned to sound uncertain in socially acceptable ways. It examines the paper’s idea of “self-knowledge” through the lens of confidence calibration, including the dangerous case where a model does not know an answer but responds with unwarranted confidence. The...

This episode explores how the 2021 paper “Hopfield Networks Is All You Need” reframes transformer attention as a modern continuous Hopfield network, connecting attention, associative memory, and similarity-based retrieval under one mathematical lens. It explains the core idea of content-addressable memory, contrasts classical binary Hopfield networks with newer differentiable versions, and shows w...

This episode explores the 2014 seq2seq paper by Sutskever and colleagues as a turning point in machine translation, asking what problem it actually solved and why it mattered at the time. It explains how deep LSTM encoder-decoder models reframed translation as end-to-end learning of \(p(\text{output}|\text{input})\), replacing hand-built phrase tables, alignment models, and decoding heuristics wit...

This episode explores the 2014–2015 breakthrough paper that introduced attention to neural machine translation, framing it as a solution to a specific flaw in early encoder-decoder models: forcing an entire source sentence into one fixed-length vector. It explains how pre-attention RNN-based seq2seq systems struggled on long or complex sentences, and how Bahdanau et al.’s “soft alignment” let the...

This episode explores the 2017 ConvS2S paper from Facebook AI Research, which argued that sequence-to-sequence models for machine translation did not need recurrence and could instead use fully convolutional encoder-decoder networks with attention. It explains the seq2seq and neural machine translation setup, clarifies that the paper replaces recurrent computation rather than attention, and breaks...

This episode explores whether large language models can estimate their chances of success before acting, update those estimates during a task, and use them to decide when to abstain from costly work. It explains why that matters for agentic systems in coding and software environments, where overconfidence can lead to wasted effort, unsafe actions, or expensive mistakes, and connects the paper to e...

This episode explores ChartNet, a 1.5 million-sample multimodal dataset designed to improve how vision-language models read and reason about charts. It explains why chart understanding is harder than OCR or captioning alone, because models must connect visual marks, axes, legends, numerical values, and language-based reasoning with high precision. The discussion places ChartNet in the context of e...

This episode explores whether language models can express uncertainty in natural language in a way that is actually calibrated and useful, rather than merely sounding cautious or confident. It focuses on the paper’s central distinction between token uncertainty and epistemic uncertainty, arguing that next-token probabilities are a poor proxy for whether a model truly knows an answer. The discussio...

This episode explores a 2026 paper proposing that multiple LLM agents should share transformer KV-cache state, not just text, so they can avoid repeatedly paying the prefill cost of rereading the same plans, critiques, and intermediate outputs. It explains the systems background behind prefix caching, vLLM’s PagedAttention, and SGLang, then focuses on why multi-agent workflows break the exact-pref...

This episode explores Paul Werbos’s 2004 review of reverse differentiation and argues that reverse-mode automatic differentiation, backpropagation, hand-coded adjoints, and adjoint circuits are largely the same core idea expressed in different technical communities. It explains the mechanics of automatic differentiation and reverse mode in clear terms, then traces how these methods diverged histor...

This episode explores how test-time compute should be allocated in large language models, using a recent study that compares parallel sampling, majority voting, shortest- and longest-trace selection, and beam-style search under a common evaluation setup. It explains the paper’s central argument that there is no single best inference-time strategy: some model families behave like short-horizon reas...

This episode explores a 2026 paper on making reasoning models faster and cheaper by shortening their chain-of-thought without sacrificing too much accuracy. It explains the paper’s core claim that short-context RL post-training can itself push models toward concise reasoning, while also unpacking the instability this creates and how the proposed Step-Level Advantage Selection method is meant to st...

This episode explores Apple’s Stochastic KV Routing approach for reducing transformer inference costs by letting some layers reuse key-value caches from earlier layers instead of storing separate caches at every depth. It explains why KV cache memory becomes a major bottleneck for long-context autoregressive decoding, and why depth-wise cache sharing is a different idea from token eviction or temp...

This episode explores how spiking neural networks differ from conventional deep networks by representing information as time-based spikes rather than continuous activations, and why that makes them both appealing and difficult to train. It breaks down the main research directions in the field, including local spike-timing-dependent plasticity, direct gradient-based training of spiking models, and...

This episode explores a survey of FPGA-based neural network accelerators for space applications, focusing on what the literature actually demonstrates rather than treating “space AI” as a single vague category. It explains why FPGAs are appealing for onboard inference, including tight control over hardware, energy efficiency, and the ability to support vision, autonomy, compression, navigation, an...

This episode explores World-R1, a post-training method for improving 3D consistency in text-to-video generation without redesigning the underlying model architecture. It explains how the approach combines reinforcement learning, pretrained 3D reconstruction critics, vision-language rewards, and camera-motion-focused prompt data to push generated videos toward more stable geometry under viewpoint c...

This episode explores the 1997 LSTM paper as a targeted fix for why standard recurrent neural networks failed on long-range dependencies: backpropagated error signals either vanish or explode when carried through many time steps. It explains how earlier training methods like BPTT and RTRL worked, why the underlying optimization problem was mathematically ill-conditioned, and how LSTM’s memory cell...

This episode explores Compressed Convolutional Attention (CCA), a new approach that pushes transformer attention itself into a shared compressed latent space instead of only compressing the KV-cache. It walks through the evolution from standard multi-head attention to MQA, GQA, and MLA, explaining why earlier efficiency methods mostly targeted decode-time memory while leaving much of the attention...