AI Post Transformers

The November 13, 2025 paper by AMD introducs Instella, a new family of fully open-source three-billion-parameter large language models (LLMs) developed by AMD and powered by their Instinct MI300X GPUs. The central focus is on advancing transparency and reproducibility in LLMs by releasing not only the model weights but also the complete training pipeline, datasets, and optimization details. Instel...

Ten different sources are used in this episode which are excerpts from academic papers and technical reports focusing on mechanistic interpretability and sparse autoencoders in language models (LLMs) and vision-language models (VLMs). This episode explores the state-of-the-art in Mechanistic Interpretability (MI), focusing on how researchers are decomposing large language models (LLMs) and multimo...

The May 21, 2024 paper introduces Vidur, a new, high-fidelity simulation framework designed to optimize the deployment and performance of Large Language Model (LLM) inference. The authors explain that experimentally optimizing LLM deployment is prohibitively expensive, requiring exploration of a vast configuration space of system parameters like parallelization strategies and batching techniques,...

The 2024 paper introduces SYMPHONY, a novel system designed to improve memory management and scheduling for Large Language Model (LLM) inference workloads, particularly those involving multi-turn interactions like chatbots and AI agents. The authors, researchers from the University of Texas-Austin and the University of Wisconsin-Madison, explain that existing LLM serving engines either waste compu...

The April 24, 2025 academic paper introduces Tempo, a novel scheduling system designed to optimize Large Language Model (LLM) serving by addressing the wide variety of Service Level Objectives (SLOs) in modern LLM applications. The authors categorize requests into three types—latency-sensitive, throughput-intensive, and collective requests—each with distinct performance requirements that existing...

We review two papers on Spectral Gap, one 2021 and another from 2025. The first source presents the Spectral Attention Network (SAN), a novel Transformer-based architecture for graph neural networks that addresses the difficulty of defining positional encodings in graphs by leveraging the full Laplacian spectrum to learn node positions. This approach, which involves a Learned Positional Encoding (...

We provide a review of the evolution of value of Page Rank to Random Walk with Random Restart and it's application to neural networks focusing on five research papers dating from the original page rank to 2025. They collectively focus on methods for learning on graphs, particularly through the use of Random Walk Neural Networks (RWNNs) and related random walk algorithms. One primary source in...

The May 20, 2024 academic paper explores the metacognitive capabilities of Large Language Models (LLMs), specifically focusing on mathematical problem-solving. The core approach involves developing a method for a powerful LLM, such as GPT-4, to identify and label mathematical questions with specific skills, which are then organized into broader, interpretable categories. This process creates a Ski...

The January 30, 2025 paper introduces LLM-AutoDiff, a novel framework for Automatic Prompt Engineering (APE) that allows for the optimization of complex Large Language Model (LLM) workflows. This framework models an entire LLM application—including multiple LLM calls, functional components like retrievers, and cyclical operations—as a directed, auto-differentiable graph. By treating textual inputs...

These two academic papers introduce novel programming models aimed at systematically optimizing complex AI systems, particularly those built using Large Language Models (LLMs). The first source presents DSPy, a framework that abstracts traditional, hard-coded LLM pipelines into parameterized, declarative modules that can be automatically optimized using a compiler and teleprompters, demonstrating...

The four papers we review dated from 1967 up to two papers in 2025 collectively discuss the mathematical properties and deep learning applications of doubly stochastic matrices, which are nonnegative matrices whose rows and columns sum to one. One paper, "Concerning Nonnegative Matrices and Doubly Stochastic Matrices," provides the foundational mathematical theory regarding the convergen...

The October 31, 2025 paper introduces Continuous Autoregressive Language Models (CALM), a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to continuous next-vector prediction. This approach compresses a chunk of multiple tokens into a single continuous vector using a high-fidelity autoencode...

These five papers from 2022 up to 2025 discuss various knowledge distillation techniques aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces Contextualization Distillation (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to...

The August 27, 2025 paper introduces Confucius, a novel multi-agent Large Language Model (LLM) framework developed by Meta for intent-driven network management in hyper-scale environments. The framework models complex management tasks as directed acyclic graphs (DAGs) and integrates LLMs with existing tools using domain-specific languages (DSLs) to enhance planning and execution. Confucius leverag...

The June 13, 2025 joint collaboration between Stanford University, Caltech and University at Buffalo introduces a novel method called CARTRIDGE for efficiently handling long text corpora in large language models, addressing the high memory cost associated with standard In-Context Learning (ICL) and its required KV cache. A CARTRIDGE is a smaller, trained KV cache representation of a corpus, which...

The November 3, 2025 paper provide an overview of the AlphaEvolve system, an AI-powered evolutionary approach for mathematical exploration and discovery, often involving large language models (LLMs) to generate and mutate programs that act as search heuristics for finding extremal mathematical objects. The first source is a GitHub repository from Google DeepMind that serves as a collection of 67 m...

The November 22, 2024 paper from UT Texas introduces AdaFlow, a novel imitation learning framework designed to improve both the efficiency and diversity of policy generation, addressing computational bottlenecks found in previous diffusion-based methods. AdaFlow utilizes flow-based generative modeling represented by ordinary differential equations (ODEs) and incorporates a variance-adaptive ODE so...

The July 13, 2025 paper " Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications" introduces a practical framework for evaluating safety risks in real-world Large Language Model (LLM) applications, arguing that current methods focusing only on foundation models are inadequate. This framework consists of two main parts: principles for developing cus...

The August 26, 2025 collaboration between the University of Washington, NVIDIA and the Allen Institute for AI paper introduces "SuperBPE: Space Travel for Language Models," introduces SuperBPE, a novel tokenization method that challenges the standard practice of limiting tokens to subword boundaries. The authors argue that conventional Byte-Pair Encoding (BPE) is inefficient because it c...

The October 28, 2025 Samsung research paper introduces zFLoRA (zero-latency fused low-rank adapter), a novel parameter-efficient fine-tuning (PEFT) method designed to address the significant inference latency overheads associated with current adapter methods like LoRA in large language models (LLMs). The core contribution is a carefully engineered fusion of adapter blocks with the base model to ac...

The October 9, 2025 paper from UT Austin paper introduces PolicySmith, a novel framework that automates the design of system policies, arguing that the traditional manual creation of heuristics by experts is becoming inefficient due to rapidly changing environments. PolicySmith leverages Large Language Models (LLMs) and evolutionary search to generate instance-optimal heuristic code that is tailor...

The June 7, 2025 UT Austin and University of British Colombia collaboration academic paper introduces MorphKV, a novel inference-time technique designed to address the excessive memory consumption caused by Key-Value (KV) caches in Large Language Models (LLMs) during extended responses. The core problem is that KV cache size grows linearly with sequence length, straining GPU memory, leading to pri...

The June 5, 2025 research paper introducing HALoS: Hierarchical Asynchronous Local SGD, a novel optimization framework designed for training large language models (LLMs) across geographically distributed accelerators and slow, high-latency networks. The core challenge addressed is the inefficiency of standard synchronous training methods due to slow inter-region communication and heterogeneous har...

These two papers (years 2017, 2022) introduce and then apply the Gumbel-Softmax distribution as a differentiable gradient estimator for categorical and discrete latent variables in neural networks. The first paper, "Categorical Reparameterization with Gumbel-Softmax," proposes this distribution to address the challenge of backpropagating through non-differentiable sampling operations, de...

The October 29 2025 Google research paper introduces Supervised Reinforcement Learning (SRL), a novel framework designed to improve the complex, multi-step reasoning abilities of large language models (LLMs). The core issue addressed is that conventional training methods like Supervised Fine-Tuning (SFT) and outcome-based Reinforcement Learning with Verifiable Rewards (RLVR) struggle with difficul...