AI Post Transformers

The September 25, 2025 collaboration between Sea AI Lab, SUTD, NUS, NTU and University of Waterloo paper proposes an alternative to traditional Reinforcement Learning (RL) for Large Language Models (LLMs) by introducing the Feedback-Conditional Policy (FCP), which learns directly from rich verbal feedback instead of compressing it into scalar rewards. The authors argue that scalarization leads to...

This 2025 CMU paper introduces LithOS, a novel operating system designed to improve the efficiency and utilization of Graphics Processing Units (GPUs) for machine learning (ML) workloads in data centers. The authors argue that current GPU management solutions, such as NVIDIA's MPS and MIG, are too coarse-grained, leading to low utilization and high latency in multi-tenant environments. LithOS...

This October 23, 2025 Xidian University academic survey systematically reviews the transformative impact of Large Language Models (LLMs) on the three core stages of Knowledge Graph (KG) construction: ontology engineering, knowledge extraction, and knowledge fusion. The text explains that LLMs are shifting the paradigm from rigid, rule-based systems to unified, adaptive, and generative frameworks....

The October 7, 2025 technical release by Liquid AI introducing their new model, LFM2-8B-A1B, an on-device Mixture-of-Experts (MoE) designed for efficiency on consumer hardware. This model boasts 8.3 billion total parameters but only uses 1.5 billion active parameters per token, allowing it to achieve larger model quality with significantly reduced compute requirements. The document highlights the...

The provided text is an academic paper titled "Active Use of Latent Constituency Representation in both Humans and Large Language Models," which explores how sentences are internally represented in both the human brain and large language models (LLMs) like ChatGPT. The authors introduce a novel one-shot learning word deletion task where participants infer a deletion rule from a single ex...

On October 20, 2025 Hugging Face released MTEB v2, a significant refactoring of the Massive Text Embedding Benchmark, which was originally designed for evaluating text embedding models across various tasks like classification and retrieval. The update addresses package bloating and the need for broader support by introducing a more consistent interface, better typing, and improved documentation. K...

This March 27, 2025 Anthropic paper provides an overview and detailed excerpts from two related Anthropic papers concerning the interpretability of large language models, specifically focusing on Claude 3.5 Haiku. The core objective is to reverse engineer the internal computational mechanisms, or "circuits," that drive the model's behavior, analogous to studying biology or neuroscie...

The October 22, 2025 GigaAI paper introduces GigaBrain-0, a novel Vision-Language-Action (VLA) model designed for general-purpose robotic systems, which is primarily trained using a combination of real-world robot data and synthetic data generated by a world model called GigaWorld. This approach aims to enhance generalization across various real-world conditions by leveraging diverse synthetic dat...

This is a classic review of a now old but yet still important paper, the original Flash Attention paper. We review this in light of advances in compiler technology. The June 23, 2022 Stanford paper describes the original FlashAttention, an innovative, IO-aware algorithm designed to significantly enhance the efficiency of the attention mechanism in Transformer models by optimizing memory usage and...

The June 2025 paper presents excerpts from a study examining the cognitive and performance differences in essay writing among participants using a Large Language Model (LLM) like ChatGPT, a traditional Search Engine, or no external tools (Brain-only). The research uses EEG connectivity analysis to illustrate that the Brain-only group experienced a higher cognitive load, characterized by stronger a...

We review the Cattell-Horn-Carroll (CHC) used in recent AI papers on the definition of what AGI could be. The provided sources offer a comprehensive overview of the Cattell–Horn–Carroll (CHC) theory of human cognitive abilities, a widely accepted psychological model of intelligence. The first source, a Wikipedia excerpt, explains that the CHC theory synthesizes two prior models, Cattell and Horn&#...

The October 10, 2025 academic paper from Google DeepMind and the University of Michigan investigates "overthinking" in large language models (LLMs), a phenomenon where models engage in excessive, inefficient reasoning for simple queries. The authors introduce a systematic analyzer called TRACE (Thought-process Reconstruction and Automated Clustering Engine) to structurally understand how...

The July 2019 paper introduces RoBERTa, a robustly optimized BERT pretraining approach, which is a refined version of the original BERT model. The authors conduct a replication study of BERT pretraining to assess the impact of various hyperparameters, finding that BERT was significantly undertrained and could be improved by simple modifications like training longer with bigger batches, removing th...

The Meta Superintelligence Labs team in collaboration with Rice University and National University of Singapore have followed up with a version 2 of their REFRAG paper on October 12, 2025, now with actual details of how they pulled off their largest RAG innovations. We had a podcast coverage of their first version of their pre-print paper where no details were given. Fortunately this new paper doe...

The October 14, 2025 paper introduxes RAG-Anything, a novel and unified framework for Retrieval-Augmented Generation (RAG) designed to overcome the limitations of existing text-only systems when processing real-world multimodal documents. The core innovation is a dual-graph construction strategy that represents diverse content—text, images, tables, and equations—as interconnected knowledge entitie...

The October 15, 2025 paper details a novel information retrieval framework called LATTICE, which uses a Large Language Model (LLM) to perform hierarchical retrieval over a large document corpus. This approach addresses the limitations of traditional retrieve-then-rerank and generative methods by organizing documents into a semantic tree structure offline, allowing the LLM to navigate the corpus wi...

The October 21, 2025 academic paper introduces LightMem, a novel and efficient memory-augmented generation framework designed to enhance Large Language Models (LLMs) in complex, long-horizon interactions. Inspired by the human Atkinson–Shiffrin model of memory, LightMem structures information into three stages: sensory memory for lightweight, rapid input filtering; topic-aware short-term memory fo...

This June 8, 2025 collaboration between University of Texas and NYU paper describes a newly identified structural inefficiency in Large Language Models (LLMs) where the self-attention mechanism in many deeper transformer layers collapses to a near rank-one structure, which the authors term "lazy layers" that are redundant and inefficient. To address this, the authors propose a novel trai...

The May 17, 2023 academic paper explores the nature of in-context learning (ICL) in neural sequence models, particularly transformers, by investigating whether they implicitly implement standard learning algorithms like linear regression without parameter updates. Theoretically, the authors demonstrate that transformers can be constructed to implement algorithms such as gradient descent and closed...

The October 12, 2025 paper introduces EssenceBench, a novel methodology for compressing large language model (LLM) benchmarks while preserving evaluation fidelity. The core problem addressed is sample redundancy in existing benchmarks like the Open LLM Leaderboard, which is quantified through both text-level redundancy (semantic overlap) and ranking-level redundancy (correlation of model performan...

The October 16, 2025 academic paper introduces Elastic-Cache, an innovative, training-free strategy designed to significantly accelerate the inference speed of diffusion large language models (DLMs) by optimizing Key-Value (KV) cache management. Standard DLMs suffer from slow decoding because they redundantly recompute the KV cache for all tokens at every step, despite minimal changes, especially...

The October 14, 2025 paper is an excerpt from a research paper introducing Dr. LLM, a novel, retrofittable framework designed to improve the efficiency and accuracy of Large Language Models (LLMs). The core problem addressed is the wasteful static processing where every input token passes through all transformer layers, which the authors solve by equipping frozen, pretrained LLMs with lightweight,...

This large collaboration between 29 different institutions proposes a quantifiable framework for defining Artificial General Intelligence (AGI), characterized as an AI matching or exceeding the versatility and proficiency of a well-educated adult. This framework utilizes the Cattell-Horn-Carroll (CHC) theory of cognitive abilities, the most empirically validated model of human intelligence, to sys...

The September 25 2025 academic paper evaluates the performance and portability of the novel Mojo programming language for high-performance computing (HPC) scientific kernels on modern GPUs. Researchers compare Mojo’s performance against vendor-specific baselines, CUDA for NVIDIA H100 and HIP for AMD MI300A GPUs, using four workloads: two memory-bound (seven-point stencil and BabelStream) and two c...

The October 10, 2025 Duke University academic paper introduces a novel geometric framework that views Large Language Model (LLM) reasoning as continuous, evolving trajectories—or flows—within the model's representation space. The core hypothesis posits that while surface semantics determine the position of these representations, the underlying logical structure acts as a local differential co...