AI Post Transformers

This March 27, 2025 Anthropic paper provides an overview and detailed excerpts from two related Anthropic papers concerning the interpretability of large language models, specifically focusing on Claude 3.5 Haiku. The core objective is to reverse engineer the internal computational mechanisms, or "circuits," that drive the model's behavior, analogous to studying biology or neuroscie...

The October 22, 2025 GigaAI paper introduces GigaBrain-0, a novel Vision-Language-Action (VLA) model designed for general-purpose robotic systems, which is primarily trained using a combination of real-world robot data and synthetic data generated by a world model called GigaWorld. This approach aims to enhance generalization across various real-world conditions by leveraging diverse synthetic dat...

This is a classic review of a now old but yet still important paper, the original Flash Attention paper. We review this in light of advances in compiler technology. The June 23, 2022 Stanford paper describes the original FlashAttention, an innovative, IO-aware algorithm designed to significantly enhance the efficiency of the attention mechanism in Transformer models by optimizing memory usage and...

The June 2025 paper presents excerpts from a study examining the cognitive and performance differences in essay writing among participants using a Large Language Model (LLM) like ChatGPT, a traditional Search Engine, or no external tools (Brain-only). The research uses EEG connectivity analysis to illustrate that the Brain-only group experienced a higher cognitive load, characterized by stronger a...

We review the Cattell-Horn-Carroll (CHC) used in recent AI papers on the definition of what AGI could be. The provided sources offer a comprehensive overview of the Cattell–Horn–Carroll (CHC) theory of human cognitive abilities, a widely accepted psychological model of intelligence. The first source, a Wikipedia excerpt, explains that the CHC theory synthesizes two prior models, Cattell and Horn&#...

The October 10, 2025 academic paper from Google DeepMind and the University of Michigan investigates "overthinking" in large language models (LLMs), a phenomenon where models engage in excessive, inefficient reasoning for simple queries. The authors introduce a systematic analyzer called TRACE (Thought-process Reconstruction and Automated Clustering Engine) to structurally understand how...

The July 2019 paper introduces RoBERTa, a robustly optimized BERT pretraining approach, which is a refined version of the original BERT model. The authors conduct a replication study of BERT pretraining to assess the impact of various hyperparameters, finding that BERT was significantly undertrained and could be improved by simple modifications like training longer with bigger batches, removing th...

The Meta Superintelligence Labs team in collaboration with Rice University and National University of Singapore have followed up with a version 2 of their REFRAG paper on October 12, 2025, now with actual details of how they pulled off their largest RAG innovations. We had a podcast coverage of their first version of their pre-print paper where no details were given. Fortunately this new paper doe...

The October 14, 2025 paper introduxes RAG-Anything, a novel and unified framework for Retrieval-Augmented Generation (RAG) designed to overcome the limitations of existing text-only systems when processing real-world multimodal documents. The core innovation is a dual-graph construction strategy that represents diverse content—text, images, tables, and equations—as interconnected knowledge entitie...

The October 15, 2025 paper details a novel information retrieval framework called LATTICE, which uses a Large Language Model (LLM) to perform hierarchical retrieval over a large document corpus. This approach addresses the limitations of traditional retrieve-then-rerank and generative methods by organizing documents into a semantic tree structure offline, allowing the LLM to navigate the corpus wi...

The October 21, 2025 academic paper introduces LightMem, a novel and efficient memory-augmented generation framework designed to enhance Large Language Models (LLMs) in complex, long-horizon interactions. Inspired by the human Atkinson–Shiffrin model of memory, LightMem structures information into three stages: sensory memory for lightweight, rapid input filtering; topic-aware short-term memory fo...

This June 8, 2025 collaboration between University of Texas and NYU paper describes a newly identified structural inefficiency in Large Language Models (LLMs) where the self-attention mechanism in many deeper transformer layers collapses to a near rank-one structure, which the authors term "lazy layers" that are redundant and inefficient. To address this, the authors propose a novel trai...

The May 17, 2023 academic paper explores the nature of in-context learning (ICL) in neural sequence models, particularly transformers, by investigating whether they implicitly implement standard learning algorithms like linear regression without parameter updates. Theoretically, the authors demonstrate that transformers can be constructed to implement algorithms such as gradient descent and closed...

The October 12, 2025 paper introduces EssenceBench, a novel methodology for compressing large language model (LLM) benchmarks while preserving evaluation fidelity. The core problem addressed is sample redundancy in existing benchmarks like the Open LLM Leaderboard, which is quantified through both text-level redundancy (semantic overlap) and ranking-level redundancy (correlation of model performan...

The October 16, 2025 academic paper introduces Elastic-Cache, an innovative, training-free strategy designed to significantly accelerate the inference speed of diffusion large language models (DLMs) by optimizing Key-Value (KV) cache management. Standard DLMs suffer from slow decoding because they redundantly recompute the KV cache for all tokens at every step, despite minimal changes, especially...

The October 14, 2025 paper is an excerpt from a research paper introducing Dr. LLM, a novel, retrofittable framework designed to improve the efficiency and accuracy of Large Language Models (LLMs). The core problem addressed is the wasteful static processing where every input token passes through all transformer layers, which the authors solve by equipping frozen, pretrained LLMs with lightweight,...

This large collaboration between 29 different institutions proposes a quantifiable framework for defining Artificial General Intelligence (AGI), characterized as an AI matching or exceeding the versatility and proficiency of a well-educated adult. This framework utilizes the Cattell-Horn-Carroll (CHC) theory of cognitive abilities, the most empirically validated model of human intelligence, to sys...

The September 25 2025 academic paper evaluates the performance and portability of the novel Mojo programming language for high-performance computing (HPC) scientific kernels on modern GPUs. Researchers compare Mojo’s performance against vendor-specific baselines, CUDA for NVIDIA H100 and HIP for AMD MI300A GPUs, using four workloads: two memory-bound (seven-point stencil and BabelStream) and two c...

The October 10, 2025 Duke University academic paper introduces a novel geometric framework that views Large Language Model (LLM) reasoning as continuous, evolving trajectories—or flows—within the model's representation space. The core hypothesis posits that while surface semantics determine the position of these representations, the underlying logical structure acts as a local differential co...

This October 15, 2025 collaboration between Meta, UT Austin, UCL, UC Berkeley, Harvard University, and Periodic Labs details a systematic study on scaling compute for reinforcement learning (RL) in large language models (LLMs), aiming to bring predictability to the RL training phase. The authors introduce a principled framework that uses a sigmoidal curve to model the relationship between compute...

These sources and patent discuss SanDisk's development of High Bandwidth Flash (HBF), a technology designed to address the significant memory and bandwidth demands of artificial intelligence models, particularly at the edge, such as on smartphones. The first article details a presentation by SanDisk's Alper Ilkbahar, who introduced HBF as a NAND-based memory solution that mimics High Ban...

This April 2021 academic paper from NVIDIA discusses the challenge of designing converged GPUs that efficiently handle the diverging architectural demands of High Performance Computing (HPC), which uses higher precision arithmetic, and Deep Learning (DL), which increasingly uses low precision math. The authors propose a new architecture called a Composable On-PAckage GPU (COPA-GPU), which uses mul...

The sources detail a novel method called MHA2MLA (Multi-Head Attention to Multi-Head Latent Attention), which efficiently adapts pre-trained large language models (LLMs) to the memory-saving Multi-head Latent Attention (MLA) architecture without requiring full retraining. This framework achieves significant Key-Value (KV) cache compression (up to 96.87% reduction in Llama2-7B) through two main com...

The October 9, 2025 paper details the architecture, training, and evaluation of UniVideo, a unified multimodal generative system capable of handling a wide array of image and video tasks. UniVideo integrates a frozen Multimodal Large Language Model (MLLM) for understanding complex instructions and a multimodal Diffusion Transformer (MMDiT) for generation, connected by a trainable MLP. The system i...

The October 9, 2025 paper from Tencent Youtu Lab introduces Training-Free Group Relative Policy Optimization (Training-Free GRPO), a novel method designed to enhance the performance of Large Language Model (LLM) agents without requiring expensive parameter updates or fine-tuning. This approach, rooted in reinforcement learning principles, shifts policy optimization from the parameter space to the...