AI Post Transformers

We review Los Alamos National Laboratory advancements in managing indirect memory accesses in high-performance computing and it's relationship to overcoming the memory wall. The first goal of DoE’s next-generation supercomputer, ATS-5, is “Overcoming the memory wall: continued memory bandwidth performance improvements for tri-lab applications. ”"DX100" introduces a programmable data...

This June 2022 paper introduces Switch Transformers, a novel architecture designed to enhance the efficiency and scalability of large-scale language models. Unlike traditional models that reuse the same parameters, Switch Transformers employ a Mixture-of-Experts (MoE) approach, activating different parameters for each input to achieve a sparsely-activated model with significantly more parameters a...

This August 2025 survey paper explores efficient architectures for large language models (LLMs), addressing the computational challenges of models like Transformers. It categorizes advancements into linear sequence modeling, including linear attention and state-space models, which offer linear computational complexity. The document also examines sparse sequence modeling, such as static and dynamic...

This August 2020 paper introduces linear transformers, a novel approach to addressing the computational and memory inefficiencies of traditional transformer models, particularly for long sequences. By reframing the self-attention mechanism using a linear dot-product of kernel feature maps, the authors reduce the computational complexity from quadratic to linear, enabling significantly faster autor...

This August 2025 paper introduces Self-Search Reinforcement Learning (SSRL), a novel method that enables Large Language Models (LLMs) to access and utilize their internal knowledge for search-driven tasks, bypassing the need for external search engines like Google or Bing. The research explores how repeated sampling can enhance an LLM's intrinsic search capabilities and investigates the impac...

This August 2025 paper offers a comprehensive overview of diffusion language models (DLMs), contrasting them with traditional autoregressive (AR) and masked language models (MLMs). It highlights DLMs' unique advantages like parallel generation and iterative refinement, which address common AR model bottlenecks. The text also covers training methodologies for DLMs, including pre-training and f...

The source analyzes Large Language Model (LLM) inference, specifically focusing on how continuous batching significantly improves efficiency compared to traditional static batching. It explains the inefficiencies of static batching where GPUs are underutilized due to varying output lengths in a batch, and introduces continuous batching (also known as dynamic batching or iteration-level scheduling)...

This April 2024 paper introduces Atom, a novel low-bit quantization method designed to enhance the efficiency and accuracy of Large Language Model (LLM) serving. The core challenge addressed is the high computational and memory costs associated with LLMs, especially when accommodating numerous user requests. Atom tackles this by quantizing both weights and activations to low-bit representations, l...

This 2022 paper is a reminder of issues with mmap() for databases. Yet many Vector Databases today rely on mmap(). This academic paper critically evaluates the use of memory-mapped file I/O (mmap) in Database Management Systems (DBMSs), arguing against its perceived benefits over traditional buffer pool implementations. The authors explain that while mmap appears to simplify file I/O by letting th...

A video transcript is used to review the PostgreSQL Development Conference (PGConf.dev 2025) presentation titled "Scaling Postgres to the Next Level at OpenAI." The speaker, Bhan from OpenAI, outlines their extensive experience scaling PostgreSQL to support critical, read-heavy workloads without traditional sharding, achieving millions of queries per second. He discusses optimizations im...

This reviews the IETF Parallel Network File System (pNFS), an extension to NFS that separates file metadata from data storage. Specifically, "RFC 8435" introduces the initial flexible file layout type, enabling pNFS to utilize existing protocols and storage devices with limited metadata server interaction, incorporating client-side mirroring for data replication. Then we review, "Pa...

We review Colossal-AI's NVMe offload functionality, designed to overcome GPU memory limitations when training large-scale models by transferring optimizer states to NVMe disks. It highlights the TensorNVMe library, which facilitates this process and is compatible with various disk types, though NVMe SSDs are recommended for optimal performance. The text further explains the pipelined optimiza...

This is a huge review of 13 different sources on advancements in GPU-accelerated computing, focusing on data access, memory management, and performance optimization for large datasets. Several sources highlight NVIDIA's initiatives like GPUDirect Storage and the AI Data Platform, which streamline data transfer directly between storage and GPUs, reducing CPU bottlenecks. Conversely, other docu...

This February 2025 paper introduces fMoE, a novel fine-grained expert offloading system designed to optimize the serving efficiency of Mixture-of-Experts (MoE) Large Language Models (LLMs). The paper highlights the memory inefficiency of current MoE-based LLMs during inference due to inactive experts residing in GPU memory and the limitations of existing coarse-grained offloading solutions that st...

This February 2025 paper introduces ELMo-Tune-V2, a novel framework that leverages Large Language Models (LLMs) to fully automate the optimization of Log-Structured Merge-tree-based Key-Value Stores (LSM-KVS). Unlike previous methods that rely on human experts or limited automated tuning, ELMo-Tune-V2 integrates LLMs for self-navigated workload characterization, automatic tuning across a broad par...

This paper February 2025 paper introduces AiSAQ (All-in-Storage ANNS with Product Quantization), a novel method designed for Approximate Nearest Neighbor Search (ANNS) that significantly reduces DRAM (Dynamic Random-Access Memory) usage. Unlike traditional methods like DiskANN, which store compressed vectors in DRAM, AiSAQ offloads these to SSD (Solid-State Drive) storage, allowing for near-zero m...

This academic paper introduces Qwen-Image, an open-source model designed for generating high-quality images from text. It details the multi-stage data filtering pipeline used to curate a diverse and high-quality training dataset, categorized into Nature, Design, People, and Synthetic Data. The paper also explains the Multimodal Scalable RoPE (MSRoPE) encoding strategy, which improves image resolut...

The provided source introduces Mem0 and Mem0g, two novel memory architectures designed to enhance Large Language Models (LLMs) by overcoming their inherent context window limitations and improving long-term conversational coherence. Mem0 focuses on dynamically extracting, consolidating, and retrieving salient information from conversations in natural language text, while Mem0g augments this with g...

This August 2025 paper from Arizona State University's Data Mining and Machine Learning Lab investigates whether Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) represents genuine inference or merely superficial pattern matching. The authors hypothesize that CoT effectiveness is bounded by the training data's distribution, proposing that LLMs generate reasoning paths by...

This document explores the challenges associated with training deep feedforward neural networks, specifically investigating why standard gradient descent with random initialization performs poorly. The authors examine the impact of various non-linear activation functions, like sigmoid, hyperbolic tangent, and a new softsign function, on network performance and the issue of unit saturation. They fu...

This reviews the paper which introduced ZeRO-Offload, a novel technology designed to democratize large-scale deep learning model training by making it accessible even with limited GPU resources. It achieves this by strategically offloading data and computations to the CPU, thereby significantly increasing the size of models that can be trained on a single GPU—up to 13 billion parameters. The paper...

This document introduces PagedAttention, an innovative attention algorithm, and vLLM, a high-throughput serving system for large language models (LLMs). The core problem addressed is the inefficient memory management of Key-Value (KV) cache in existing LLM serving systems, leading to significant memory waste and limited batch sizes. Inspired by operating system virtual memory and paging techniques...

The provided text describes TierTrain: Proactive Memory Tiering for CPU-Based DNN Training, a paper presented at the International Symposium on Memory Management (ISMM 2025), co-located with PLDI 2025 in Seoul, South Korea. Authored by researchers from Intel Labs, the paper introduces TierTrain, a novel memory tiering solution designed to optimize Deep Neural Network (DNN) training on CPUs. This s...

The provided sources discuss advancements in large language models (LLMs), specifically focusing on test-time compute scaling to enhance reasoning performance. One paper introduces s1-32B, an open-source model trained on a small, curated dataset of 1,000 reasoning problems, and its novel technique called budget forcing. This method controls the model's "thinking time" to improve acc...

This document provides a comprehensive overview of differentiable programming, a paradigm enabling gradient-based optimization of computer programs, even those with complex control flows and data structures. It explores the fundamental concepts from automatic differentiation, including Jacobian and Hessian matrices, to the mathematical representation of programs as computation graphs and chains, e...