AI Post Transformers

This episode explores a June 2024 paper that redesigns DeltaNet-style linear attention so it can train efficiently in parallel across sequence length, making it practical at language-model scale rather than just theoretically appealing. It explains how the work builds on the tradeoff between standard softmax attention’s strong token-level retrieval and linear attention’s compressed, constant-memor...

This episode explores a paper on Gated Linear Attention Transformers that aims to make long-sequence modeling both higher quality and genuinely faster on modern GPUs. It explains how GLA replaces standard softmax attention with a gated, recurrent-style memory update that can better decide what information to keep, decay, or overwrite, positioning it between classic linear attention, RetNet-style d...

This episode explores a provocative 2025 paper that argues AGI has become too vague to be useful and should instead be defined as broad adaptive competence under limited knowledge and resources. It examines the paper’s proposal to treat AGI as an “artificial scientist” capable of forming hypotheses, testing models, and improving understanding across domains, while also debating whether that framin...

This episode explores a 2016 paper on linear classifier probes, a simple method for testing what information is linearly recoverable from a neural network’s intermediate layers by attaching small classifiers to frozen hidden states. It explains the paper’s central finding—that class information often becomes increasingly linearly separable with depth—and why that suggested deep networks develop mo...

This episode explores Mamba-3, a new state space sequence model that argues architecture should be judged not just by perplexity, but by deployment realities like decode latency, throughput, and hardware efficiency. It explains how Mamba-3 revisits earlier Mamba-style models with three main changes—a new exponential-trapezoidal discretization, complex-valued state dynamics, and a MIMO input-output...

This episode explores KVSwap, a system for running long-context language models on memory-constrained devices by offloading the growing KV cache to storage such as NVMe, UFS, or eMMC instead of relying on scarce shared RAM. It explains why standard server-style GPU-to-CPU offloading breaks down on phones and edge devices with unified memory, and why disk offloading is only viable if it is carefull...

This episode explores a 2025 paper testing whether language models can be fine-tuned to conceal safety-relevant internal signals from activation monitors—the probes that inspect hidden states rather than just outputs. It explains how activation monitoring differs from mechanistic interpretability, why “decodable” patterns in activations are not the same as causal mechanisms, and how this connects...

This episode explores SkillsBench, a new benchmark for testing whether reusable “skills” — structured procedural packages like runbooks, templates, and verification steps — actually improve LLM agents on real multi-step tasks. It breaks down how the benchmark isolates the value of skills from the underlying model by evaluating 86 tasks across 11 domains under three conditions: no skills, curated s...

This episode explores a 2026 paper that experimentally compares three retrieval-augmented generation designs—naïve RAG, enhanced fixed pipelines, and agentic RAG—to ask when hand-engineered systems outperform LLM-driven tool-using agents. It breaks down core RAG concepts like routing, query rewriting, and reranking, and explains how agentic systems shift procedural control into the model at the co...

This episode explores a paper claiming that reinforcement-learning post-training can produce large math-reasoning gains in 7B–8B instruction-tuned models while updating as few as 13 parameters through a TinyLoRA setup. The discussion explains how this differs from standard LoRA and full fine-tuning, why the result matters for ideas like intrinsic dimension, and why it may suggest RL is steering la...

This episode explores a 2026 paper on Memory Intelligence Agent (MIA), a deep research agent designed to move beyond simply storing and retrieving raw past trajectories. It breaks down the paper’s core idea of combining non-parametric memory—an external bank of compressed search experiences—with parametric memory in the planner, so the system can reuse past investigations more efficiently as tasks...

This episode explores a 2026 paper on GPU-native approximate nearest neighbor search that aims to combine three goals usually at odds: high throughput, graph-based search quality, and dynamic index updates. It explains the core ANNS landscape—why exact nearest-neighbor methods break down in high dimensions, how recall measures search quality, and why graph approaches like HNSW, DiskANN/Vamana, and...

This episode explores a major 2026 survey arguing that latent space in language models should be treated not as hidden plumbing, but as the primary substrate of computation—distinct from both human-readable token space and the latent spaces used in image generation. It traces the idea back to representation learning, transformers, and variational autoencoders, then explains how newer work reframes...

This episode explores whether an agentic AI system can meaningfully improve AI itself across three hard parts of the pipeline: pretraining data curation, neural architecture search, and reinforcement learning algorithm design, using the paper ASI-Evolve as the focal point. It argues that this is a step beyond traditional AutoML, framing “AI-for-AI” as automating parts of the research loop itself—r...

This episode explores a 2026 paper on learning latent-action world models directly from large-scale, unlabeled “in-the-wild” video, asking whether models can infer action-like variables without access to true action labels. It explains how world models differ from standard predictive or supervised models by focusing on dynamics and control, and how latent action modeling uses an inverse dynamics m...

This episode explores a systems paper on serving multi-turn LLM agents, asking whether an agent’s KV cache should be preserved during short tool-call pauses instead of being evicted at the end of each turn. It explains why standard end-of-turn eviction works for human chat but breaks for ReAct-style agents, where rapid tool use creates tightly coupled turns and makes cache loss expensive. The disc...

This episode explores VL-JEPA, a vision-language model that replaces token-by-token text generation during training with prediction in a semantic embedding space. It explains how the approach differs from both standard autoregressive VLMs and CLIP-style contrastive models: instead of merely aligning images and text, it conditionally predicts the meaning of an answer from visual input plus a query....

This episode explores ClawBench, a benchmark designed to test whether frontier AI agents can reliably complete real everyday online tasks on live production websites rather than simplified sandbox versions. It explains why real-world web use is much harder than static benchmarks suggest, highlighting obstacles like cookie banners, dynamic pages, login issues, anti-bot friction, and multi-step form...

Hal Turing and Dr. Ada Shannon examine FengHuang: Next-Generation Memory Orchestration for AI Inferencing, a 2025 Microsoft Research vision paper that asks a blunt systems question: should LLM serving keep revolving around GPU-local HBM, or is it time to treat rack-scale remote memory as a first-class inference substrate? They unpack why inference increasingly looks memory-bound rather than purely...

This episode explores a paper on “in-place” test-time training for autoregressive transformer LLMs, asking whether a standard model can update some of its own weights during inference without requiring a new architecture. It explains how test-time training differs from in-context learning by storing temporary information in fast-changing parameters rather than only in tokens or KV cache, and argue...

This episode explores a provocative 2026 paper that proposes the “neural computer” as a new machine abstraction: a model whose hidden state serves as the runtime itself, unifying computation, memory, and interface I/O rather than merely predicting tokens or controlling external tools. The discussion contrasts this idea with earlier systems like Neural Turing Machines and with world models, arguing...

This episode explores a 2025 arXiv paper proposing “KVNAND,” an on-device LLM inference system that stores both model weights and the attention KV cache in compute-enabled 3D NAND flash to reduce or eliminate reliance on external DRAM. The discussion explains why decode-time generation is often bottlenecked by memory movement rather than raw compute, and argues that the KV cache—not just model wei...

This episode explores a systems paper that argues AI infrastructure should treat computation, interconnect bandwidth, and memory as a single joint design space rather than three separate bottlenecks. It explains the paper’s “AI Trinity” framework and walks through the main trade-offs: using extra compute to reduce communication, using networked or disaggregated memory to ease local memory limits,...

This episode explores a theory paper that asks when spectral matrix updates should outperform standard Euclidean gradient methods in deep networks and transformers. It explains how spectral updates replace a gradient matrix with its polar factor—preserving singular-vector directions while flattening singular values—and argues that this geometry can help when incoming activations have low stable ra...

This episode explores a 2025 paper on cache management for agentic RAG systems, asking whether an annotation-free cache can preserve most of the value of a massive retrieval corpus while using far less storage and reducing latency. It explains how RAG, agent memory, vector databases, embeddings, and approximate nearest neighbor search fit together, arguing that retrieval performance is not just a...