AI Post Transformers

This episode examines the fundamental latency bottleneck in autoregressive language models: sequential token generation requires one full transformer forward pass per output token, leaving GPU parallelism idle during single-user inference. The episode centers on Draxler et al. (5 co-authors, UC Irvine and Chan-Zuckerberg Initiative), whose paper on Parallel Token Prediction landed Christmas Eve 20...

On the February 19, 2026 paper Google Deepmind introduces Unified Latents (UL), a novel framework for generative modeling that jointly trains an encoder, a diffusion prior, and a diffusion decoder. By incorporating a fixed amount of Gaussian noise during the encoding process, the method creates a stable and interpretable bound on latent information. This architecture allows for precise control ove...

QuantSpec is a novel self-speculative decoding framework designed to accelerate the inference of Large Language Models, particularly in long-context scenarios. The system addresses memory and latency bottlenecks by employing a hierarchical 4-bit quantized KV cache and quantized weights, allowing a draft model to share the same architecture as the target model. This approach maintains a high token...

In a collaboration between Google DeepMind, University of Texas at Austin, University of Washington and Harvard published on December 2024 researchers introduce MatFormer, a novel elastic Transformer architecture designed to improve the efficiency of large-scale foundation models. Unlike traditional models that require independent training for different sizes, this framework allows a single univer...

We review an April 3, 2025 research collaboration between CMU, Moffett AI and Together AI which introduces MagicDec, a new framework designed to accelerate the serving of long-context large language models through speculative decoding. Previously, conventional wisdom discouraged using speculative decoding (SD) for large batches, as the verification step was believed to be too compute-heavy and ine...

We review three different papers which focus on different KV cache optimizations techniques using different KV selection algorithms types: static vs dynamic. StreamingLLM and SnapKV use static KV selection methods. The dynamic strategy KV selection is introduced in PQCache. These different KV selection algorithms explore different KV budgets and speculation lengths to estimate optimal theoretical...

These sources review historically speculative decoding, an innovative technique designed to accelerate Large Language Model (LLM) inference without reducing output quality. Large models are traditionally slow because they generate text one token at a time, a process limited by hardware memory bandwidth. To solve this, a much smaller and faster approximation model suggests multiple future tokens in...

The provided documents describe the development and evolution of EAGLE, a high-efficiency framework designed to accelerate Large Language Model (LLM) inference through speculative sampling. By performing autoregression at the feature level rather than the token level and incorporating shifted token sequences to manage sampling uncertainty, the original EAGLE achieves significant speedups while mai...

The researchers introduce CXL-SpecKV, a specialized architecture designed to overcome the memory bottlenecks of large language model serving by offloading key-value caches to remote memory. By utilizing Compute Express Link (CXL) and FPGA accelerators, the system enables memory disaggregation, which expands available storage capacity by up to eight times compared to standard GPU setups. A core inn...

This article outlines how Baseten optimized speculative decoding using the TensorRT-LLM framework to accelerate model inference. The authors detail overcoming technical hurdles such as inefficient batching, hardware contention, and server instability to make the technique viable for production environments. By synchronizing the execution of draft and target models and patching core software bugs,...

Speculative Streaming is a novel inference method designed to accelerate large language model (LLM) generation without the need for traditional auxiliary "draft" models. By integrating multi-stream attention directly into the target model, the system can perform future n-gram prediction and token verification simultaneously within a single forward pass. This approach eliminates t...

Apple researchers have introduced on December 2025 Mirror Speculative Decoding (Mirror-SD), an advanced inference algorithm designed to accelerate large language models by overcoming the sequential bottlenecks of standard decoding. Traditional methods are often limited by the time it takes for a small draft model to suggest tokens before a larger target model can verify them. Mirror-SD breaks this...

We review two papers which examine the integration of speculative decoding and request batching to accelerate Large Language Model (LLM) inference. While both techniques aim to improve GPU hardware utilization, the research identifies a critical tension where high batch sizes can actually diminish the effectiveness of speculation. To resolve this, the authors propose adaptive strategies that dynam...

On a paper published January 21, 2026 researchers from MIT and NVIDIA explain how they have have developed a new system called Taming the Long Tail (TLT) to solve computational inefficiencies in training reasoning-heavy large language models. During standard training, many processors sit idle while waiting for the longest text sequences to finish generating, creating a massive resource bottleneck....

These sources explore advanced techniques for accelerating Large Language Model (LLM) inference through speculative decoding, a process where smaller "draft" models predict tokens for a larger "target" model to verify in parallel. A primary focus is Multi-Draft Speculative Decoding (MDSD), which uses multiple draft sequences to increase the probability of acceptance and reduce...

MEDUSA is a novel framework introduced on June 24 2024 designed to accelerate Large Language Model (LLM) inference by overcoming the delays caused by sequential token generation. Instead of relying on a separate draft model like traditional speculative decoding, it incorporates multiple decoding heads that predict several subsequent tokens simultaneously. These predictions are organized into a tre...

The February 12.2026 research from the University of Virginia and Google introduces the deep-thinking ratio (DTR), a novel metric designed to measure the true reasoning effort of large language models by analyzing internal token stabilization. While traditional metrics like token count often fail to predict accuracy due to "overthinking," DTR tracks how many layers a model requires befor...

Researchers from METR introduce a novel framework for evaluating AI progress by measuring a model's time horizon, defined as the length of a task a human can complete that an AI can perform with 50% reliability. Traditional benchmarks often fail because they saturate quickly or focus on static knowledge, whereas this approach uses economically valuable tasks in fields like software engineerin...

The September 26 2025 research paper introduces FastGRPO, a high-efficiency framework designed to accelerate the training of large language models using Group Relative Policy Optimization. The authors identify that the generation phase is the primary bottleneck in reinforcement learning, accounting for over 90% of total training time. To solve this, they implement concurrency-aware speculative dec...

This research collaboration between King’s College London, Google DeepMind on a research paper published on February 19, 2026 introduces a novel framework for evaluating the collective behavior of large language model (LLM) agents within complex social dilemmas. By prompting models to generate high-level algorithmic strategies rather than individual actions, the authors successfully simulated inte...

Researchers introduced on May 2024 self-speculative decoding, a novel "plug-and-play" inference scheme designed to accelerate Large Language Models (LLMs) without requiring auxiliary models or extra memory. This method utilizes a two-stage process where a faster, lower-quality draft is generated by selectively skipping intermediate layers of the original model. These draft tokens are the...

These sources detail advanced reinforcement learning frameworks designed to improve how quadruped robots navigate difficult, real-world environments. The first source introduces a single-stage teacher-student method that utilizes skeleton information and a system-response model to achieve more natural, stable movement. The second source proposes ZSL-RPPO, a zero-shot learning architecture that eli...

The provided sources explore advanced techniques for optimizing large language model (LLM) inference, specifically by addressing the memory bottlenecks of the Key-Value (KV) cache. KVQuant introduces a high-precision quantization framework that utilizes per-channel scaling, non-uniform datatypes, and sparse outlier handling to compress activations to sub-4-bit precision with minimal accuracy loss....

The Deepmind February 3, 2023 paper "Accelerating Large Language Model Decoding with Speculative Sampling introduced speculative sampling, a novel algorithm designed to increase the speed of Large Language Model (LLM) decoding without altering the final output. The researchers utilize a smaller, faster draft model to predict multiple potential tokens, which are then verified in parallel by a...

The January 26, 2026 Stanford research paper introduces Agentic Plan Caching (APC), a novel framework designed to reduce the high operational costs of Large Language Model (LLM) agents. Traditional caching methods often fail because agent workflows are highly dynamic and dependent on external environments, making simple input-output storage ineffective. The APC framework solves this by extracting...