AI Post Transformers

The May 24, 2025 UT Austin paper introduces the Anchored Diffusion Language Model (ADLM), a novel approach that aims to improve discrete language modeling by addressing the limitations of traditional Autoregressive (AR) and standard Diffusion Language Models (DLMs). AR models, such as GPT-3, generate text sequentially and struggle with complex reasoning, while existing DLMs, which use iterative ma...

The August 9, 2023 paper introduces the Retentive Network (RetNet), a proposed foundational architecture for large language models intended to succeed the Transformer model. RetNet aims to overcome the Transformer's inefficiencies during inference by simultaneously achieving training parallelism, low-cost inference, and strong performance, a combination previously considered an "impossib...

The October 30, 2025 technical report details the development and evaluation of Kimi Linear, a novel hybrid linear attention architecture for large language models (LLMs). The core innovation is the Kimi Delta Attention (KDA) module, which refines existing linear attention mechanisms to achieve superior performance and efficiency compared to traditional full attention, particularly in long-context...

The April 22, 2022 collaboration between University of Washington, Facebook AI and the Allen Institute for AI introduces Attention with Linear Biases (ALiBi), a novel and efficient method for position representation in transformer models that effectively addresses the challenge of extrapolation—a model's ability to maintain performance on input sequences longer than those used during training...

The October 24, 2025 collaboration between many universities have published a paper thst compares the performance of Large Language Models (LLMs) and Small Language Models (SLMs) on requirements classification tasks within software engineering. Researchers conducted a preliminary study using eight models across three datasets to address concerns about the high computational cost and privacy risks...

The August 26, 2024 academic paper introduces Quest, a novel algorithm designed to improve the inference efficiency of Long-Context Large Language Models (LLMs) by addressing the costly self-attention process caused by a large Key-Value (KV) cache. Quest utilizes Query-Aware Sparsity to dynamically identify and select only the critical KV cache pages based on the current query token, which signifi...

The June 5, 2025 collaboration between University of Edinburgh and Nvidia paper introduces the concept of inference-time hyper-scaling for large language models (LLMs), which aims to boost reasoning accuracy by allowing for longer or more parallel token sequences within the same computational budget. The core bottleneck is identified as the size of the key–value (KV) cache, which grows linearly an...

The September 19, 2025 Alibaba paper introduces Flash-LLM, a novel software framework designed to enable cost-effective and highly-efficient inference for large generative models by supporting unstructured sparsity on high-performance tensor cores. The authors observe that the primary bottleneck in large language model (LLM) inference is the memory bandwidth limitation during "skinny" ma...

The February 12, 2025 KuaiShou Inc paper introduces ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression framework designed to address the scalability issues of traditional transformer-based sequential recommender systems, which suffer from quadratic complexity with respect to sequence length. ELASTIC achieves this by proposing a Linear Dispatcher Attention (LDA) layer that c...

The October 21, 2025 collaboration paper between UW-Madison and Amazon Web Services discuss the critical role of the Multi-Layer Perceptron (MLP) intermediate size f_size as the primary architectural component for introducing non-linearity and complexity within Large Language Models (LLMs). The MLP layer achieves this by taking the hidden state d_model projecting it up to the expanded f_size, appl...

On October 29, 2025 Anthropic presented research investigating the existence of functional introspective awareness in large language models (LLMs), specifically focusing on Anthropic's Claude models. The core methodology involves using concept injection, where researchers manipulate a model's internal activations with representations of specific concepts to see if the model can accuratel...

The source provides excerpts from a scientific paper introducing TxGNN, a novel graph foundation model designed for zero-shot drug repurposing, which aims to identify therapeutic candidates even for diseases with no existing treatments or limited molecular data. Developed by researchers affiliated with institutions like Harvard Medical School and Stanford University, this model leverages a medical...

The provided text introduces Sentence-BERT (SBERT), a modification of the popular BERT and RoBERTa language models, designed to efficiently generate semantically meaningful sentence embeddings. The authors address the significant computational overhead of using standard BERT for tasks requiring sentence-pair comparisons, such as semantic similarity search and clustering, which can take hours for l...

We cover two new innovations from Microsoft extending ideas from the original old FlashAttention. Flash Attention is an IO-aware attention algorithm for Transformers designed to address the quadratic time and memory complexity of standard self-attention on long sequences. By using tiling and recomputation to minimize slow High Bandwidth Memory (HBM) accesses in favor of fast on-chip SRAM, FlashAtt...

The October 20, 2025 Meta FAIR paper introduces the Free Transformer, an innovative extension of the decoder-only Transformer architecture, which addresses the limitations of purely autoregressive language modeling by integrating random latent variables into the generative process. This new model is structured as a conditional Variational Autoencoder (VAE), where an encoder learns the latent varia...

The October 23, 2025 collaboration between UC San Diego , NVIDIA , META , UW-Madison , and UNC introduces Real Deep Research (RDR), a systematic framework designed to analyze vast amounts of research literature in rapidly growing fields such as AI and robotics. The methodology uses large language and multimodal models (LLMs/LMMs) for content extraction, reasoning, and semantic embedding to map the...

The October 23 2025 research paper probes the spatial reasoning capabilities of Large Language Models (LLMs) when processing text-based inputs, specifically focusing on how performance degrades as task complexity increases. Using a suite of five grid-based tasks—including quadrant identification, geometric transformations, distance evaluation, word searches, and tile sliding—the authors tested fou...

The October 10, 2025 paper from the University of Michigan and Google DeepMind concerning the phenomenon of "overthinking" in Large Language Models (LLMs) that utilize chain-of-thought (CoT) reasoning. The authors introduce a systematic analyzer called TRACE to structurally examine an LLM's thought process, decomposing it into sub-thoughts and progression graphs to move beyond super...

The August 26, 2025 collaboration between Stanford, NVIDIA, Shanghai Jiao Tong University, University of Michigan, University of Colorado Boulder, Carnegie Mellon University introduces Strata, a hierarchical context caching framework designed to improve the performance of serving Large Language Models (LLMs) with long context windows. The core problem Strata addresses is that while caching key-val...

These April 29, 2024 paper provides an overview of the challenges associated with using NVIDIA's Multi-Instance GPU (MIG) technology, specifically focusing on the address translation mechanism in the A100 GPU. The papers reveal, primarily through reverse-engineering efforts, that the L2 and L3 Translation Lookaside Buffers (TLBs) utilize a compression design where each entry comprises 16 sub-...

This October 23, 2025 technical report from the Ling Team introduces the Ring-linear model series, specifically Ring-mini-linear-2.0 and Ring-flash-linear-2.0, which utilize a hybrid attention architecture combining linear and softmax attention mechanisms to enhance efficiency in long-context reasoning. The paper explains how this architecture, featuring Mixture-of-Experts (MoE) and advanced FP8 t...

The October 25, 2025 Bytedance paper introduces Open-o3 Video, a novel framework developed by researchers from Peking University and ByteDance, aimed at advancing video reasoning by incorporating explicit spatio-temporal evidence. Unlike prior models that only generate textual rationales, Open-o3 Video explicitly highlights key timestamps and bounding boxes to ground its answers in visual observat...

The September 26, 2025 paper introduces a novel reinforcement learning framework called Meta-Awareness via Self-Alignment (MASA), designed to enhance the reasoning capabilities and efficiency of large language models (LLMs) by improving their meta-awareness, or the ability to know "how to think." MASA works by creating parallel rollouts for both solution paths and meta-predictions (like...

The October 3, 2025 paper by Tencent introduces a reinforcement learning technique called Low-probability Regularization (Lp-Reg) designed to overcome the exploration collapse bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models. The authors identify that performance plateaus because training systematically eliminates crucial, low-probability tokens, termed...

The September 25, 2025 collaboration between Sea AI Lab, SUTD, NUS, NTU and University of Waterloo paper proposes an alternative to traditional Reinforcement Learning (RL) for Large Language Models (LLMs) by introducing the Feedback-Conditional Policy (FCP), which learns directly from rich verbal feedback instead of compressing it into scalar rewards. The authors argue that scalarization leads to...