AI Post Transformers

The four papers we review dated from 1967 up to two papers in 2025 collectively discuss the mathematical properties and deep learning applications of doubly stochastic matrices, which are nonnegative matrices whose rows and columns sum to one. One paper, "Concerning Nonnegative Matrices and Doubly Stochastic Matrices," provides the foundational mathematical theory regarding the convergen...

The October 31, 2025 paper introduces Continuous Autoregressive Language Models (CALM), a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to continuous next-vector prediction. This approach compresses a chunk of multiple tokens into a single continuous vector using a high-fidelity autoencode...

These five papers from 2022 up to 2025 discuss various knowledge distillation techniques aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces Contextualization Distillation (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to...

The August 27, 2025 paper introduces Confucius, a novel multi-agent Large Language Model (LLM) framework developed by Meta for intent-driven network management in hyper-scale environments. The framework models complex management tasks as directed acyclic graphs (DAGs) and integrates LLMs with existing tools using domain-specific languages (DSLs) to enhance planning and execution. Confucius leverag...

The June 13, 2025 joint collaboration between Stanford University, Caltech and University at Buffalo introduces a novel method called CARTRIDGE for efficiently handling long text corpora in large language models, addressing the high memory cost associated with standard In-Context Learning (ICL) and its required KV cache. A CARTRIDGE is a smaller, trained KV cache representation of a corpus, which...

The November 3, 2025 paper provide an overview of the AlphaEvolve system, an AI-powered evolutionary approach for mathematical exploration and discovery, often involving large language models (LLMs) to generate and mutate programs that act as search heuristics for finding extremal mathematical objects. The first source is a GitHub repository from Google DeepMind that serves as a collection of 67 m...

The November 22, 2024 paper from UT Texas introduces AdaFlow, a novel imitation learning framework designed to improve both the efficiency and diversity of policy generation, addressing computational bottlenecks found in previous diffusion-based methods. AdaFlow utilizes flow-based generative modeling represented by ordinary differential equations (ODEs) and incorporates a variance-adaptive ODE so...

The July 13, 2025 paper " Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications" introduces a practical framework for evaluating safety risks in real-world Large Language Model (LLM) applications, arguing that current methods focusing only on foundation models are inadequate. This framework consists of two main parts: principles for developing cus...

The August 26, 2025 collaboration between the University of Washington, NVIDIA and the Allen Institute for AI paper introduces "SuperBPE: Space Travel for Language Models," introduces SuperBPE, a novel tokenization method that challenges the standard practice of limiting tokens to subword boundaries. The authors argue that conventional Byte-Pair Encoding (BPE) is inefficient because it c...

The October 28, 2025 Samsung research paper introduces zFLoRA (zero-latency fused low-rank adapter), a novel parameter-efficient fine-tuning (PEFT) method designed to address the significant inference latency overheads associated with current adapter methods like LoRA in large language models (LLMs). The core contribution is a carefully engineered fusion of adapter blocks with the base model to ac...

The October 9, 2025 paper from UT Austin paper introduces PolicySmith, a novel framework that automates the design of system policies, arguing that the traditional manual creation of heuristics by experts is becoming inefficient due to rapidly changing environments. PolicySmith leverages Large Language Models (LLMs) and evolutionary search to generate instance-optimal heuristic code that is tailor...

The June 7, 2025 UT Austin and University of British Colombia collaboration academic paper introduces MorphKV, a novel inference-time technique designed to address the excessive memory consumption caused by Key-Value (KV) caches in Large Language Models (LLMs) during extended responses. The core problem is that KV cache size grows linearly with sequence length, straining GPU memory, leading to pri...

The June 5, 2025 research paper introducing HALoS: Hierarchical Asynchronous Local SGD, a novel optimization framework designed for training large language models (LLMs) across geographically distributed accelerators and slow, high-latency networks. The core challenge addressed is the inefficiency of standard synchronous training methods due to slow inter-region communication and heterogeneous har...

These two papers (years 2017, 2022) introduce and then apply the Gumbel-Softmax distribution as a differentiable gradient estimator for categorical and discrete latent variables in neural networks. The first paper, "Categorical Reparameterization with Gumbel-Softmax," proposes this distribution to address the challenge of backpropagating through non-differentiable sampling operations, de...

The October 29 2025 Google research paper introduces Supervised Reinforcement Learning (SRL), a novel framework designed to improve the complex, multi-step reasoning abilities of large language models (LLMs). The core issue addressed is that conventional training methods like Supervised Fine-Tuning (SFT) and outcome-based Reinforcement Learning with Verifiable Rewards (RLVR) struggle with difficul...

The May 24, 2025 UT Austin paper introduces the Anchored Diffusion Language Model (ADLM), a novel approach that aims to improve discrete language modeling by addressing the limitations of traditional Autoregressive (AR) and standard Diffusion Language Models (DLMs). AR models, such as GPT-3, generate text sequentially and struggle with complex reasoning, while existing DLMs, which use iterative ma...

The August 9, 2023 paper introduces the Retentive Network (RetNet), a proposed foundational architecture for large language models intended to succeed the Transformer model. RetNet aims to overcome the Transformer's inefficiencies during inference by simultaneously achieving training parallelism, low-cost inference, and strong performance, a combination previously considered an "impossib...

The October 30, 2025 technical report details the development and evaluation of Kimi Linear, a novel hybrid linear attention architecture for large language models (LLMs). The core innovation is the Kimi Delta Attention (KDA) module, which refines existing linear attention mechanisms to achieve superior performance and efficiency compared to traditional full attention, particularly in long-context...

The April 22, 2022 collaboration between University of Washington, Facebook AI and the Allen Institute for AI introduces Attention with Linear Biases (ALiBi), a novel and efficient method for position representation in transformer models that effectively addresses the challenge of extrapolation—a model's ability to maintain performance on input sequences longer than those used during training...

The October 24, 2025 collaboration between many universities have published a paper thst compares the performance of Large Language Models (LLMs) and Small Language Models (SLMs) on requirements classification tasks within software engineering. Researchers conducted a preliminary study using eight models across three datasets to address concerns about the high computational cost and privacy risks...

The August 26, 2024 academic paper introduces Quest, a novel algorithm designed to improve the inference efficiency of Long-Context Large Language Models (LLMs) by addressing the costly self-attention process caused by a large Key-Value (KV) cache. Quest utilizes Query-Aware Sparsity to dynamically identify and select only the critical KV cache pages based on the current query token, which signifi...

The June 5, 2025 collaboration between University of Edinburgh and Nvidia paper introduces the concept of inference-time hyper-scaling for large language models (LLMs), which aims to boost reasoning accuracy by allowing for longer or more parallel token sequences within the same computational budget. The core bottleneck is identified as the size of the key–value (KV) cache, which grows linearly an...

The September 19, 2025 Alibaba paper introduces Flash-LLM, a novel software framework designed to enable cost-effective and highly-efficient inference for large generative models by supporting unstructured sparsity on high-performance tensor cores. The authors observe that the primary bottleneck in large language model (LLM) inference is the memory bandwidth limitation during "skinny" ma...

The February 12, 2025 KuaiShou Inc paper introduces ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression framework designed to address the scalability issues of traditional transformer-based sequential recommender systems, which suffer from quadratic complexity with respect to sequence length. ELASTIC achieves this by proposing a Linear Dispatcher Attention (LDA) layer that c...

The October 21, 2025 collaboration paper between UW-Madison and Amazon Web Services discuss the critical role of the Multi-Layer Perceptron (MLP) intermediate size f_size as the primary architectural component for introducing non-linearity and complexity within Large Language Models (LLMs). The MLP layer achieves this by taking the hidden state d_model projecting it up to the expanded f_size, appl...