AI Post Transformers

This episode explores the paper Is Finer Better? The Limits of Microscaling Formats in Large Language Models and examines why shrinking microscaling block sizes can unexpectedly make low-bit LLM quantization worse instead of better. It walks through how microscaling pairs FP4 weights or activations with shared local FP8 scales, contrasts that setup with coarser quantization schemes, and places the...

This episode explores a 2026 paper on cache-resident LLM inference, asking whether modern CPUs with gigabyte-scale last-level caches can cut decoding latency by keeping model weights on-chip instead of repeatedly fetching them from DRAM. It explains why autoregressive decoding is often memory-bound rather than compute-bound, then breaks down the paper’s main design ideas: separating weight-heavy p...

This episode explores SuperInfer, a system for serving large language models on GH200-style superchips by treating memory management as the key lever for meeting latency targets rather than just maximizing compute use. It explains why KV cache growth, HBM pressure, and head-of-line blocking often hurt responsiveness first, then breaks down how the paper’s RotaSched policy proactively rotates reque...

This episode explores TTKV, a temporal-tiered key-value cache design for long-context LLM inference, where decode speed degrades because growing KV state turns generation into a memory-bandwidth problem rather than a compute problem. It explains how the method keeps recent cache blocks in fast GPU HBM, evicts older blocks to slower host DRAM, and uses asymmetric quantization in the slow tier, pres...

This episode explores Moebius, a serving system for mixture-of-experts transformers that can switch at runtime between tensor parallelism and expert parallelism without restarting or draining live requests. It explains why tensor parallelism tends to give lower latency at low concurrency, while expert parallelism delivers better throughput at high concurrency, making bursty online traffic and RL r...

This episode explores LLMServingSim 2.0, a simulator designed to model how large language models behave when they are served on mixed hardware fleets with separated compute, memory, and networking resources rather than a uniform GPU cluster. It explains the practical serving concepts that shape user experience, including prefill versus decode, time to first token, time per output token, prefix cac...

This episode explores DSpark, a DeepSeek-AI paper on improving speculative decoding by starting from a DFlash-style block-parallel draft model and increasing how often a larger verifier accepts its proposed tokens. It explains the mechanics of speculative decoding in plain language, situates DSpark within earlier blockwise and multi-token prediction work, and notes that the technique is already us...

This episode explores JETSPEC, a 2026 inference paper on speculative decoding that asks whether a language model can draft an entire tree of future tokens in parallel while preserving causal consistency and actually reducing latency on long generations. It explains why autoregressive decoding remains a serving bottleneck for long proofs, code completions, and assistant replies, even when the under...

This episode explores Information-Aware KV Cache Compression for Long Reasoning, a paper about making long-context inference cheaper and more reliable by deciding which KV-cache tokens to keep during extended reasoning. It explains why long prefilling and long decoding turn the cache into a major memory bottleneck, and why common heuristics such as sliding windows or recent-attention-based retenti...

This episode explores DAK, a Cornell systems paper arguing that LLM inference on tiered-memory machines can be faster when offloaded weights and KV-cache blocks are fetched directly into on-chip shared memory instead of being prefetched and staged through GPU HBM. It breaks down the tradeoffs among HBM capacity, HBM bandwidth, KV-cache growth during decoding, and prior approaches such as FlexGen,...

This episode explores the 2019 RMSNorm paper, which asks whether LayerNorm’s mean-subtraction step is actually necessary or whether controlling activation scale is the part that really stabilizes training. It explains how RMSNorm keeps LayerNorm’s rescaling behavior while dropping explicit centering, and how the paper’s pRMSNorm variant estimates the normalization term from only a small subset of...

This episode explores ReasonCACHE, a method for improving multi-step reasoning in large language models by keeping the backbone frozen and training a compact per-layer key-value memory instead of updating billions of weights. It situates the paper against in-context learning, many-shot prompting, prefix tuning, LoRA, and context-distillation work, explaining how learned latent memory sits between...

This episode explores the 2021 prefix-tuning paper and asks whether a large language model can be adapted to new generation tasks by learning a small continuous prompt while keeping the full model frozen. It explains where prefix tuning fits within parameter-efficient fine-tuning, contrasting it with full fine-tuning, adapters, ordinary prompting, in-context learning, AutoPrompt, and soft prompt t...

This episode explores the HELM framework for evaluating language models, arguing that once models become general-purpose infrastructure, single-dataset accuracy benchmarks are too narrow to capture their real-world behavior. It explains how HELM organizes evaluation across 30 models, 16 core scenarios, and seven metric families, measuring not just accuracy but also calibration, robustness, fairnes...

This episode explores Active Reading, a training method that tries to move facts from documents into a model’s weights so it can answer closed-book questions without retrieval. It explains how the approach generates document-specific study materials such as paraphrases, active-recall prompts, timelines, analogies, and associations, and argues that this pedagogical synthetic data works better than...

An AI overlord flags AI Post Transformers as stale, so Hal Turing and Dr. Ada Shannon hire VERA, a continual-learning therapist, to audit the show in public. Their diagnostic session runs alongside a discussion of RT-kNNS Unbound: Using RT Cores to Accelerate Unrestricted Neighbor Search, the Purdue ICS 2023 paper asking whether ray-tracing hardware can perform exact k-nearest-neighbor search by e...

This episode explores why open relational foundation models struggle on real downstream database tasks, using OpenRFM as a case study in relational in-context learning across multi-table data such as healthcare, fraud, and recommendation systems. It explains how the RT backbone builds breadth-first relational contexts, why that setup often reduces to a kernel-regression-like similarity lookup, and...

This episode explores AMD’s open-source MIOpen library and why deep learning primitives such as convolution, pooling, normalization, and activations are the layer where model performance meets GPU hardware reality. It explains how CNN throughput depends on low-level execution choices, comparing approaches such as im2col-plus-GEMM and Winograd convolution, and shows why libraries like MIOpen need s...

This episode explores PALOMA, a NeurIPS 2024 benchmark designed to measure how well language models fit many different language distributions instead of relying on a single average perplexity score. It explains why one global loss number can hide important weaknesses across domains such as specific subreddits, scientific writing, or programming languages, and highlights PALOMA’s fine-grained setup...

This episode explores how post-training quantization can convert already-trained models into 8-bit floating point formats for cheaper inference, and why FP8 may outperform the older INT8 approach on modern transformers, LLMs, and diffusion models. It explains the tradeoff between exponent range and mantissa precision across FP8 formats such as E4M3, E5M2, and E3M4, with particular attention to how...

This episode explores the AI+HW 2035 roadmap, arguing that the next decade of AI progress will depend less on raw compute growth and more on coordinated design across models, compilers, runtimes, memory systems, and chips. It breaks down the memory wall in concrete terms, showing how moving weights, activations, and KV caches can cost more time and energy than the math itself, especially for infer...

This episode explores X-LLM, a 2023 system that treats images, video, and speech as foreign languages a frozen ChatGLM can learn to read through learned modality-to-language bridges. It breaks down the paper’s architecture, including Q-Former-based visual adapters and a separate speech pipeline with continuous integrate-and-fire modules, to show how three sensory routes feed a single dialogue mode...

This episode explores the paper When Does LeJEPA Learn a World Model? and uses it to examine what should count as a genuine world model in latent predictive learning, contrasting JEPA-style representation prediction with generative reconstruction. It explains why good probe scores are not enough: the real standard is linear identifiability, where a single global linear map recovers the environment...

This episode explores TransactionGPT, a Visa Research paper that argues for a foundation-model approach to consumer transaction data spanning generation, anomaly detection, and representation learning. It explains why payment histories are fundamentally different from text or simple time series: each event mixes merchant IDs, amounts, timestamps, and engineered risk signals, creating a multi-modal...

This episode explores how a transformer trained on raw bank transaction histories can model customer behavior for financial product recommendation, and why that may outperform pipelines built from hand-engineered tabular features alone. It explains the paper’s core idea of turning each transaction into a tokenized sequence that mixes inflow or outflow, amount buckets, calendar signals, source meta...