AI Post Transformers

This episode explores whether multi-agent systems can benefit from test-time scaling in the same way single models do, focusing on a 2025 paper that combines learned collaborative reasoning with runtime orchestration. It explains the paper’s core setup: a model trained on 500 carefully curated multi-agent reasoning traces (M500) and a separate “CEO” controller that coordinates specialized agents s...

This episode explores TUMIX, a test-time scaling framework that turns a single strong language model into a team of specialized agents with different tool-use strategies, including plain-text reasoning, code execution, search, and hybrids. It explains the paper’s core argument that better reasoning may come not from simply sampling one model more times, but from diversifying computational pathways...

This episode explores a paper that tests whether general LLM agents remain effective when search, coding, reasoning, and API/tool-use tasks are mixed together under one shared prompt, interface, and tool set rather than optimized benchmark-specific setups. It explains how the benchmark is built by unifying tasks from BrowseComp, WebVoyager, SWE-Bench Verified, Terminal-Bench, MathHay, Tau2-Bench,...

This episode explores a 2026 paper on AgentArk, which asks whether the reasoning gains of multi-agent LLM systems can be compressed into a single model, reducing the latency, token cost, and orchestration burden of running a “committee” of models at inference time. It explains multi-agent systems as setups where multiple model instances debate, critique, and revise one another, arguing that their...

This episode explores a systems paper on making multi-agent LLM setups far more efficient by sharing most of the KV cache across agents that use the same base model with different LoRA adapters. It explains the core argument: for a shared long context, the backbone model’s hidden states are nearly identical across agents, while most role-specific differences come from LoRA’s low-rank adapter outpu...

This episode explores a systems paper on speeding up retrieval-augmented generation by reusing KV caches for frequently repeated retrieved documents, even when those documents are not exact prompt prefixes. It explains why long RAG prompts make prefill the main latency bottleneck, why standard prefix caching only helps in narrow cases, and why naive non-prefix cache reuse can hurt quality by ignor...

This episode explores the Qwen3.5-Omni technical report as a significant update in omnimodal AI: a system designed to understand and generate text, audio, images, and video within one architecture. It unpacks the model’s Thinker–Talker design, arguing that separating multimodal reasoning from real-time output is especially important for speech, where latency and timing make generation far harder t...

This episode explores Gated Delta Networks, a sequence-modeling approach that combines Mamba-style gating with DeltaNet-style selective memory updates to improve long-context and retrieval-heavy performance. It explains how linear attention and state-space models compress the past into a fixed recurrent state, why that makes them hardware-efficient, and where they often fail: memory collisions tha...

This episode explores whether newer hybrid-attention language models make prefill-decode disaggregation practical across clusters or even datacenters by shrinking the KV cache enough to move it over ordinary Ethernet. It explains why the real production bottleneck is not request routing but transferring the attention state between prefill and decode, and contrasts dense transformers—where KV cache...

This episode explores a philosophical challenge to computational functionalism through Alexander Lerchner’s “The Abstraction Fallacy,” which argues that software can simulate conscious behavior without ever producing real subjective experience. It examines whether computation is an objective physical process or an interpretation imposed on physical systems, contrasting Lerchner’s “mapmaker” idea w...

This episode explores NVIDIA’s Nemotron 3 Super, an open 120B-parameter model with only 12B active parameters per token, and examines whether its hybrid Mamba-transformer Mixture-of-Experts design can approach frontier-model accuracy while delivering much better inference efficiency. The discussion breaks down the paper’s main ingredients—MoE sparsity, LatentMoE for more practical serving, Mamba-s...

This episode explores a systems paper on speeding up LLM “Re-Prefill,” the step where a model reloads a previously saved shared prefix KV cache from CPU or SSD, computes a request-specific suffix, and produces the first token. It explains why prefix KV reuse is valuable for shared-context workloads like RAG, conversational search, and multi-turn document QA, but argues that offloading creates new...

This episode explores a June 2024 paper that redesigns DeltaNet-style linear attention so it can train efficiently in parallel across sequence length, making it practical at language-model scale rather than just theoretically appealing. It explains how the work builds on the tradeoff between standard softmax attention’s strong token-level retrieval and linear attention’s compressed, constant-memor...

This episode explores a paper on Gated Linear Attention Transformers that aims to make long-sequence modeling both higher quality and genuinely faster on modern GPUs. It explains how GLA replaces standard softmax attention with a gated, recurrent-style memory update that can better decide what information to keep, decay, or overwrite, positioning it between classic linear attention, RetNet-style d...

This episode explores a provocative 2025 paper that argues AGI has become too vague to be useful and should instead be defined as broad adaptive competence under limited knowledge and resources. It examines the paper’s proposal to treat AGI as an “artificial scientist” capable of forming hypotheses, testing models, and improving understanding across domains, while also debating whether that framin...

This episode explores a 2016 paper on linear classifier probes, a simple method for testing what information is linearly recoverable from a neural network’s intermediate layers by attaching small classifiers to frozen hidden states. It explains the paper’s central finding—that class information often becomes increasingly linearly separable with depth—and why that suggested deep networks develop mo...

This episode explores Mamba-3, a new state space sequence model that argues architecture should be judged not just by perplexity, but by deployment realities like decode latency, throughput, and hardware efficiency. It explains how Mamba-3 revisits earlier Mamba-style models with three main changes—a new exponential-trapezoidal discretization, complex-valued state dynamics, and a MIMO input-output...

This episode explores KVSwap, a system for running long-context language models on memory-constrained devices by offloading the growing KV cache to storage such as NVMe, UFS, or eMMC instead of relying on scarce shared RAM. It explains why standard server-style GPU-to-CPU offloading breaks down on phones and edge devices with unified memory, and why disk offloading is only viable if it is carefull...

This episode explores a 2025 paper testing whether language models can be fine-tuned to conceal safety-relevant internal signals from activation monitors—the probes that inspect hidden states rather than just outputs. It explains how activation monitoring differs from mechanistic interpretability, why “decodable” patterns in activations are not the same as causal mechanisms, and how this connects...

This episode explores SkillsBench, a new benchmark for testing whether reusable “skills” — structured procedural packages like runbooks, templates, and verification steps — actually improve LLM agents on real multi-step tasks. It breaks down how the benchmark isolates the value of skills from the underlying model by evaluating 86 tasks across 11 domains under three conditions: no skills, curated s...

This episode explores a 2026 paper that experimentally compares three retrieval-augmented generation designs—naïve RAG, enhanced fixed pipelines, and agentic RAG—to ask when hand-engineered systems outperform LLM-driven tool-using agents. It breaks down core RAG concepts like routing, query rewriting, and reranking, and explains how agentic systems shift procedural control into the model at the co...

This episode explores a paper claiming that reinforcement-learning post-training can produce large math-reasoning gains in 7B–8B instruction-tuned models while updating as few as 13 parameters through a TinyLoRA setup. The discussion explains how this differs from standard LoRA and full fine-tuning, why the result matters for ideas like intrinsic dimension, and why it may suggest RL is steering la...

This episode explores a 2026 paper on Memory Intelligence Agent (MIA), a deep research agent designed to move beyond simply storing and retrieving raw past trajectories. It breaks down the paper’s core idea of combining non-parametric memory—an external bank of compressed search experiences—with parametric memory in the planner, so the system can reuse past investigations more efficiently as tasks...

This episode explores a 2026 paper on GPU-native approximate nearest neighbor search that aims to combine three goals usually at odds: high throughput, graph-based search quality, and dynamic index updates. It explains the core ANNS landscape—why exact nearest-neighbor methods break down in high dimensions, how recall measures search quality, and why graph approaches like HNSW, DiskANN/Vamana, and...

This episode explores a major 2026 survey arguing that latent space in language models should be treated not as hidden plumbing, but as the primary substrate of computation—distinct from both human-readable token space and the latent spaces used in image generation. It traces the idea back to representation learning, transformers, and variational autoencoders, then explains how newer work reframes...