AI Post Transformers

This May 2025 paper introduces FastVLM, an innovative approach designed to enhance the efficiency of Vision Language Models (VLMs). The authors explain that while increasing image resolution is crucial for VLM performance, traditional visual encoders become inefficient. FastVLM addresses this by incorporating FastViTHD, a novel hybrid vision encoder that reduces both the number of visual tokens an...

This paper introduces Apertus, a large language model developed by the Swiss AI Initiative, a partnership between ETH Zurich and EPFL. The GitHub repository appears to host technical documentation or code related to Apertus, while the Hugging Face page provides a comprehensive overview of the Apertus-8B-Instruct-2509 model. This model is highlighted for being fully open, massively multilingual (su...

This August 2025 paper offers an extensive overview of the evolution and application of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) within scientific research, primarily focusing on the period from 2018 to 2025. It details how these AI models have progressed through various paradigm shifts, from initial transfer learning to sophisticated scientific agents capable of a...

This August 2025 paper introduces rStar2-Agent, a 14B math reasoning model developed by Microsoft Research that achieves state-of-the-art performance comparable to much larger models by employing agentic reinforcement learning. The model is trained to "think smarter" through three key innovations: an efficient RL infrastructure that manages high-throughput code execution, a novel GRPO-Ro...

This September 2024 paper introduces FusionANNS, a novel system designed to improve Approximate Nearest Neighbor Search (ANNS) for extremely large datasets. It addresses challenges in existing ANNS systems, such as performance bottlenecks, high operational costs, and accuracy limitations, particularly when dealing with billion-scale vector data in modern AI infrastructure like Large Language Model...

This academic paper addresses the inherent challenges in training Recurrent Neural Networks (RNNs), specifically the vanishing and exploding gradient problems. The authors explore these issues from analytical, geometrical, and dynamical systems perspectives, building upon previous work. They propose and empirically validate a gradient norm clipping strategy to combat exploding gradients and a soft...

This February 2025 research addresses the critical issue of training instability in Large Language Models (LLMs), which often stems from sudden, massive "gradient spikes" that can be thousands of times larger than typical gradients. The authors introduce Spike-Aware Adam with Momentum Reset (SPAM), a novel optimizer designed to counteract these spikes through periodic momentum resets and...

This August 2025 paper introduces Pimba, a novel Processing-in-Memory (PIM) accelerator designed to enhance the efficiency of Large Language Model (LLM) serving for both traditional transformer-based models and emerging post-transformer architectures. The authors highlight that memory bandwidth is a critical bottleneck for both types of LLMs, specifically during attention operations in transformer...

This August 2025 paper introduces Oaken, a novel acceleration solution for serving Large Language Models (LLMs) that addresses the significant challenges of memory bandwidth and capacity bottlenecks inherent in batched LLM inference. Oaken achieves this through a co-designed algorithm and hardware architecture, featuring an online-offline hybrid KV cache quantization technique. This technique effi...

This January 2019 academic paper addresses the common issue of poor generalization in adaptive gradient optimization methods like Adam, compared to traditional Stochastic Gradient Descent (SGD) with momentum. The authors demonstrate that L2 regularization and weight decay are not equivalent for adaptive optimizers, unlike for standard SGD, leading to suboptimal performance in Adam. They propose a...

This April 2018 paper introduces Adafactor, a novel optimization method designed to reduce the memory footprint of adaptive learning rate algorithms like Adam, particularly for large neural networks. Adafactor achieves this by estimating per-parameter second moments using factored representations, specifically maintaining only row and column sums for weight matrices, thereby reducing memory requir...

This August 2025 academic paper explores the application of post-training quantization (PTQ) to diffusion large language models (dLLMs), a promising alternative to traditional autoregressive LLMs for natural language generation. The authors conduct a systematic study to understand how existing PTQ techniques, commonly used for compressing AR LLMs, perform with dLLMs. A key finding is the prevalenc...

This August 2025 paper introduces ODYSSEY, a comprehensive framework for open-world mobile manipulation that integrates robotic mobility, manipulation, and real-time perception. It highlights a novel approach that uses large language models for high-level task planning and vision-language models for fine-grained action guidance, enabling robots to adaptively interact in complex environments. A sig...

This August 2025 paper presents Google's comprehensive methodology for measuring the environmental impact of AI inference workloads in a large-scale production environment. It addresses a critical gap in existing research by accounting for the full stack of AI serving infrastructure, including active AI accelerator power, host system energy, idle machine capacity, and data center overhead. Th...

This August 2025 paper introduces ComoRAG, a novel framework designed to enhance long-context narrative comprehension in Large Language Models (LLMs) by simulating human metacognitive regulation. It addresses the limitations of existing Retrieval-Augmented Generation (RAG) methods, which struggle with stateful reasoning and integrating contradictory evidence over extended narratives. ComoRAG emplo...

This August 2025 academic paper, titled "Has GPT-5 Achieved Spatial Intelligence? An Empirical Study," examines the spatial understanding and reasoning capabilities of advanced multi-modal AI models, including the recently released GPT-5. The authors propose a new taxonomy for spatial tasks and evaluate both proprietary and open-source models against eight key benchmarks, utilizing over...

This is a review of DeepSeek's latest release announced on Hugging Face on August 21, 2025. The source introduces DeepSeek-V3.1, a hybrid large language model that supports both "thinking" and "non-thinking" operational modes, distinguishable through different chat templates. This updated model offers smarter tool calling capabilities and improved thinking efficiency, prov...

This March 2025 paper introduces compressed experts, an innovative method to enhance the efficiency of Mixture-of-Experts (MoE) models by reducing computational overhead while preserving performance. The core idea involves replacing less critical "auxiliary experts" with lightweight, compact representations, called compressed experts, during fine-tuning. This strategy allows for a signif...

The source provides an overview of Google DeepMind's AI research and models, highlighting various applications across different scientific disciplines and creative fields. It introduces Genie 3, a general-purpose world model capable of generating diverse, interactive, real-time environments from text prompts. The document details Genie 3's capabilities, such as simulating physical proper...

We review Los Alamos National Laboratory advancements in managing indirect memory accesses in high-performance computing and it's relationship to overcoming the memory wall. The first goal of DoE’s next-generation supercomputer, ATS-5, is “Overcoming the memory wall: continued memory bandwidth performance improvements for tri-lab applications. ”"DX100" introduces a programmable data...

This June 2022 paper introduces Switch Transformers, a novel architecture designed to enhance the efficiency and scalability of large-scale language models. Unlike traditional models that reuse the same parameters, Switch Transformers employ a Mixture-of-Experts (MoE) approach, activating different parameters for each input to achieve a sparsely-activated model with significantly more parameters a...

This August 2025 survey paper explores efficient architectures for large language models (LLMs), addressing the computational challenges of models like Transformers. It categorizes advancements into linear sequence modeling, including linear attention and state-space models, which offer linear computational complexity. The document also examines sparse sequence modeling, such as static and dynamic...

This August 2020 paper introduces linear transformers, a novel approach to addressing the computational and memory inefficiencies of traditional transformer models, particularly for long sequences. By reframing the self-attention mechanism using a linear dot-product of kernel feature maps, the authors reduce the computational complexity from quadratic to linear, enabling significantly faster autor...

This August 2025 paper introduces Self-Search Reinforcement Learning (SSRL), a novel method that enables Large Language Models (LLMs) to access and utilize their internal knowledge for search-driven tasks, bypassing the need for external search engines like Google or Bing. The research explores how repeated sampling can enhance an LLM's intrinsic search capabilities and investigates the impac...

This August 2025 paper offers a comprehensive overview of diffusion language models (DLMs), contrasting them with traditional autoregressive (AR) and masked language models (MLMs). It highlights DLMs' unique advantages like parallel generation and iterative refinement, which address common AR model bottlenecks. The text also covers training methodologies for DLMs, including pre-training and f...