AI Post Transformers

This document explores the Mamba architecture, a novel approach to sequence modeling that offers an efficient alternative to Transformers. It primarily investigates the role of "input selectivity" within Mamba's core component, the S6 layer, and its impact on the model's capabilities. The research proves Mamba's superiority over its predecessor, S4D, in approximating disco...

This paper introduces DeepSeek-R1, a new suite of large language models developed by DeepSeek-AI, focusing on enhancing reasoning capabilities through reinforcement learning (RL). It details the development of DeepSeek-R1-Zero, a model trained purely with RL that demonstrates strong reasoning but has readability issues, and DeepSeek-R1, which addresses these flaws by incorporating multi-stage trai...

This paper introduces DeepSeek-V3, a large Mixture-of-Experts (MoE) model designed to advance open-source language model capabilities with improved training efficiency and performance. The document details its innovative architecture, including an auxiliary-loss-free load balancing strategy and a Multi-Token Prediction objective for enhanced data efficiency and future token prediction. It further...

The provided text introduces DeepSeekMoE, an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in large language models. The authors propose two key strategies: fine-grained expert segmentation, which divides experts into smaller, more numerous units for flexible combinations, and shared expert isolation, which designates specific experts for common knowled...

This reviews a document dated January 27, 2025, from Daniel and Michael at Unsloth, details their work on quantizing DeepSeek-R1's 671B parameter model, significantly reducing its size by 80% to 131GB while maintaining functionality. They achieved this dynamic quantization by selectively applying higher bitrates to crucial layers and lower bitrates to less sensitive MoE layers, contrasting wi...

Five different sources are reviewed to understand Concept Drift in neural networks. 1) https://www.nature.com/articles/s41467-024-46142-w - Empirical data drift detection experiments on real-world medical imaging data 2) https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2024.1330258/full - One or two things we know about concept drift—a survey on monitoring in evol...

We reviewed two papers on ColBERT. We review and expand upon ColBERT, a neural information retrieval model that utilizes contextualized late interaction over BERT to estimate relevance between queries and documents. The first source, the original ColBERT paper, details its architecture, which encodes queries and documents into multi-vector representations and computes relevance by summing the maxi...

This source introduces CODEGEN, a family of large language models developed by Salesforce Research, designed for program synthesis. The models, varying in size up to 16.1B parameters, are trained on extensive natural language and programming language datasets, and the training library, JAXFORMER, is open-sourced to promote accessibility. A key contribution is the exploration of multi-turn program...

This reviews two papers on Chain of Thought: 1) https://arxiv.org/pdf/2201.11903 - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models 2) https://arxiv.org/pdf/2503.11926 - Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation These papers introduce Chain-of-Thought (CoT) prompting as a method to improve the reasoning abilities of large language model...

The provided text, titled "AI and Memory Wall," examines the growing disparity between computational power and memory bandwidth in AI, particularly for Large Language Models (LLMs). It highlights how server hardware FLOPS (floating-point operations per second) have dramatically outpaced DRAM (Dynamic Random-Access Memory) and interconnect bandwidth growth over the past two decades, leadi...

The paper "Decoupled Weight Decay Regularization" by Ilya Loshchilov and Frank Hutter (2017) introduced what we know in Pytorch now as AdamW. This academic paper explores the differences between L2 regularization and weight decay regularization in optimizing neural networks, particularly focusing on adaptive gradient algorithms like Adam. The authors demonstrate that while these two regu...

This reviews the 2015 paper which introduced Adam, an algorithm for first-order gradient-based optimization in scenarios involving stochastic objective functions. Adam uniquely computes adaptive learning rates for different parameters by estimating the first and second moments of gradients, offering advantages in computational efficiency and memory requirements. The paper details Adam's algor...

This research paper explores the scaling behavior of Transformer architectures, offering insights into pre-training and fine-tuning efficiency. It challenges previous findings by demonstrating that model shape, not just size, significantly impacts downstream task performance, unlike its lesser effect on upstream pre-training loss. The study also reveals that scaling protocols vary in effectiveness...

This 2000 paper, titled "Scaling Laws for Neural Language Models," explores the empirical relationships between the performance of neural language models (specifically Transformers) and various scaling factors: model size (parameters), dataset size (tokens), and computational budget (compute used for training). The authors demonstrate that model performance follows predictable power-law...

What ResNet introduced is adding the input of a block directly to its output, like this: Output = 𝐹(𝑥)+ 𝑥 This academic paper introduces Deep Residual Learning, a novel framework designed to facilitate the training of exceptionally deep neural networks for image recognition. The core innovation lies in reformulating layers to learn residual functions, meaning they learn the difference from the inp...

This paper introduces RoFormer, an enhanced Transformer model that leverages Rotary Position Embedding (RoPE) to improve natural language processing tasks. The authors explore existing methods for incorporating positional information into Transformer architectures, contrasting traditional additive position encoding with their novel multiplicative approach. RoPE encodes absolute position through a...

This 2000 paper introduces a novel solution to a weakness found in Long Short-Term Memory (LSTM) networks, specifically when processing continuous data streams without predefined segmentation. The core problem addressed is the unbounded growth of internal cell states within standard LSTM networks, which can lead to performance degradation. The authors propose and implement "forget gates"...

This paper introduces Longformer, a novel Transformer-based model designed to overcome the limitations of traditional Transformers in processing exceptionally long sequences. Unlike prior models with quadratic scaling, Longformer employs an attention mechanism that scales linearly with sequence length, making it efficient for documents containing thousands of tokens. This innovative architecture c...

This academic paper introduces Linformer, a novel approach to address the computational bottleneck of Transformer models in natural language processing. The authors demonstrate that the self-attention mechanism, which is a core component of Transformers and typically incurs quadratic time and space complexity with respect to sequence length, can be approximated by a low-rank matrix. By exploiting...

This paper outlines the advancements in Optical Character Recognition (OCR), particularly focusing on handwritten character and word recognition using Neural Networks. The authors, affiliated with AT&T Labs-Research, detail various machine learning techniques, including Gradient-Based Learning and Convolutional Neural Networks (CNNs) like LeNet-5, highlighting their effectiveness in handling h...

The first source introduces Layer Normalization (LN), a technique designed to accelerate and stabilize the training of deep neural networks, particularly recurrent neural networks, by normalizing summed inputs across a layer for each training case. It contrasts LN with Batch Normalization, highlighting LN's independence from mini-batch size and its consistent computation during both training...

This 2022 paper explores the significant negative impact of repeated data on the performance of large language models, even when such repetitions constitute a small fraction of the total training data. The authors observe a "double descent" phenomenon, where model performance initially improves, then degrades at a specific repetition frequency, and finally improves again with excessive r...

This 2023 paper, GPT-4 Technical Report from OpenAI introduces GPT-4, a multimodal AI model capable of processing both image and text inputs to produce text outputs, demonstrating human-level performance on various professional and academic benchmarks, such as the bar exam. The report highlights the predictable scaling of the model's performance and its improved factual accuracy and adherence...

This paper introduces Grouped-Query Attention (GQA), a novel approach designed to enhance the inference efficiency of large language models. It addresses the limitations of Multi-Query Attention (MQA), which, while fast, can compromise model quality, and Multi-Head Attention (MHA), which offers high quality but is slower due to memory bandwidth overhead. The authors propose uptraining existing MHA...

This 2020 paper outlines the development and evaluation of GPT-3, a large language model, exploring its performance across various natural language processing tasks under zero-shot, one-shot, and few-shot learning conditions, which involve providing minimal to no task-specific examples during inference. It details the model's architecture, training methodology, including its use of a massive...