AI Post Transformers

We review two papers on Spectral Gap, one 2021 and another from 2025. The first source presents the Spectral Attention Network (SAN), a novel Transformer-based architecture for graph neural networks that addresses the difficulty of defining positional encodings in graphs by leveraging the full Laplacian spectrum to learn node positions. This approach, which involves a Learned Positional Encoding (...

We provide a review of the evolution of value of Page Rank to Random Walk with Random Restart and it's application to neural networks focusing on five research papers dating from the original page rank to 2025. They collectively focus on methods for learning on graphs, particularly through the use of Random Walk Neural Networks (RWNNs) and related random walk algorithms. One primary source in...

The May 20, 2024 academic paper explores the metacognitive capabilities of Large Language Models (LLMs), specifically focusing on mathematical problem-solving. The core approach involves developing a method for a powerful LLM, such as GPT-4, to identify and label mathematical questions with specific skills, which are then organized into broader, interpretable categories. This process creates a Ski...

The January 30, 2025 paper introduces LLM-AutoDiff, a novel framework for Automatic Prompt Engineering (APE) that allows for the optimization of complex Large Language Model (LLM) workflows. This framework models an entire LLM application—including multiple LLM calls, functional components like retrievers, and cyclical operations—as a directed, auto-differentiable graph. By treating textual inputs...

These two academic papers introduce novel programming models aimed at systematically optimizing complex AI systems, particularly those built using Large Language Models (LLMs). The first source presents DSPy, a framework that abstracts traditional, hard-coded LLM pipelines into parameterized, declarative modules that can be automatically optimized using a compiler and teleprompters, demonstrating...

The four papers we review dated from 1967 up to two papers in 2025 collectively discuss the mathematical properties and deep learning applications of doubly stochastic matrices, which are nonnegative matrices whose rows and columns sum to one. One paper, "Concerning Nonnegative Matrices and Doubly Stochastic Matrices," provides the foundational mathematical theory regarding the convergen...

The October 31, 2025 paper introduces Continuous Autoregressive Language Models (CALM), a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to continuous next-vector prediction. This approach compresses a chunk of multiple tokens into a single continuous vector using a high-fidelity autoencode...

These five papers from 2022 up to 2025 discuss various knowledge distillation techniques aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces Contextualization Distillation (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to...

The August 27, 2025 paper introduces Confucius, a novel multi-agent Large Language Model (LLM) framework developed by Meta for intent-driven network management in hyper-scale environments. The framework models complex management tasks as directed acyclic graphs (DAGs) and integrates LLMs with existing tools using domain-specific languages (DSLs) to enhance planning and execution. Confucius leverag...

The June 13, 2025 joint collaboration between Stanford University, Caltech and University at Buffalo introduces a novel method called CARTRIDGE for efficiently handling long text corpora in large language models, addressing the high memory cost associated with standard In-Context Learning (ICL) and its required KV cache. A CARTRIDGE is a smaller, trained KV cache representation of a corpus, which...

The November 3, 2025 paper provide an overview of the AlphaEvolve system, an AI-powered evolutionary approach for mathematical exploration and discovery, often involving large language models (LLMs) to generate and mutate programs that act as search heuristics for finding extremal mathematical objects. The first source is a GitHub repository from Google DeepMind that serves as a collection of 67 m...

The November 22, 2024 paper from UT Texas introduces AdaFlow, a novel imitation learning framework designed to improve both the efficiency and diversity of policy generation, addressing computational bottlenecks found in previous diffusion-based methods. AdaFlow utilizes flow-based generative modeling represented by ordinary differential equations (ODEs) and incorporates a variance-adaptive ODE so...

The July 13, 2025 paper " Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications" introduces a practical framework for evaluating safety risks in real-world Large Language Model (LLM) applications, arguing that current methods focusing only on foundation models are inadequate. This framework consists of two main parts: principles for developing cus...

The August 26, 2025 collaboration between the University of Washington, NVIDIA and the Allen Institute for AI paper introduces "SuperBPE: Space Travel for Language Models," introduces SuperBPE, a novel tokenization method that challenges the standard practice of limiting tokens to subword boundaries. The authors argue that conventional Byte-Pair Encoding (BPE) is inefficient because it c...

The October 28, 2025 Samsung research paper introduces zFLoRA (zero-latency fused low-rank adapter), a novel parameter-efficient fine-tuning (PEFT) method designed to address the significant inference latency overheads associated with current adapter methods like LoRA in large language models (LLMs). The core contribution is a carefully engineered fusion of adapter blocks with the base model to ac...

The October 9, 2025 paper from UT Austin paper introduces PolicySmith, a novel framework that automates the design of system policies, arguing that the traditional manual creation of heuristics by experts is becoming inefficient due to rapidly changing environments. PolicySmith leverages Large Language Models (LLMs) and evolutionary search to generate instance-optimal heuristic code that is tailor...

The June 7, 2025 UT Austin and University of British Colombia collaboration academic paper introduces MorphKV, a novel inference-time technique designed to address the excessive memory consumption caused by Key-Value (KV) caches in Large Language Models (LLMs) during extended responses. The core problem is that KV cache size grows linearly with sequence length, straining GPU memory, leading to pri...

The June 5, 2025 research paper introducing HALoS: Hierarchical Asynchronous Local SGD, a novel optimization framework designed for training large language models (LLMs) across geographically distributed accelerators and slow, high-latency networks. The core challenge addressed is the inefficiency of standard synchronous training methods due to slow inter-region communication and heterogeneous har...

These two papers (years 2017, 2022) introduce and then apply the Gumbel-Softmax distribution as a differentiable gradient estimator for categorical and discrete latent variables in neural networks. The first paper, "Categorical Reparameterization with Gumbel-Softmax," proposes this distribution to address the challenge of backpropagating through non-differentiable sampling operations, de...

The October 29 2025 Google research paper introduces Supervised Reinforcement Learning (SRL), a novel framework designed to improve the complex, multi-step reasoning abilities of large language models (LLMs). The core issue addressed is that conventional training methods like Supervised Fine-Tuning (SFT) and outcome-based Reinforcement Learning with Verifiable Rewards (RLVR) struggle with difficul...

The May 24, 2025 UT Austin paper introduces the Anchored Diffusion Language Model (ADLM), a novel approach that aims to improve discrete language modeling by addressing the limitations of traditional Autoregressive (AR) and standard Diffusion Language Models (DLMs). AR models, such as GPT-3, generate text sequentially and struggle with complex reasoning, while existing DLMs, which use iterative ma...

The August 9, 2023 paper introduces the Retentive Network (RetNet), a proposed foundational architecture for large language models intended to succeed the Transformer model. RetNet aims to overcome the Transformer's inefficiencies during inference by simultaneously achieving training parallelism, low-cost inference, and strong performance, a combination previously considered an "impossib...

The October 30, 2025 technical report details the development and evaluation of Kimi Linear, a novel hybrid linear attention architecture for large language models (LLMs). The core innovation is the Kimi Delta Attention (KDA) module, which refines existing linear attention mechanisms to achieve superior performance and efficiency compared to traditional full attention, particularly in long-context...

The April 22, 2022 collaboration between University of Washington, Facebook AI and the Allen Institute for AI introduces Attention with Linear Biases (ALiBi), a novel and efficient method for position representation in transformer models that effectively addresses the challenge of extrapolation—a model's ability to maintain performance on input sequences longer than those used during training...

The October 24, 2025 collaboration between many universities have published a paper thst compares the performance of Large Language Models (LLMs) and Small Language Models (SLMs) on requirements classification tasks within software engineering. Researchers conducted a preliminary study using eight models across three datasets to address concerns about the high computational cost and privacy risks...