AI Post Transformers

The paper introduces Dynamic Tanh (DyT), a novel element-wise operation designed to replace normalization layers in Transformer models. Traditionally, normalization layers like Layer Normalization (LN) are considered essential for stable and effective training of deep neural networks. However, this paper challenges that belief by demonstrating that DyT, which mimics the S-shaped input-output mappi...

The provided text introduces DroidSpeak, a novel distributed Large Language Model (LLM) inference system designed to enhance the efficiency of compound AI systems. It addresses the challenge of reusing Key-Value (KV) caches across different LLMs that share the same architectural foundation, a problem current systems struggle with. DroidSpeak achieves significant throughput improvements and faster...

This academic paper introduces DiMSUM, a novel architecture for image generation that enhances diffusion models by integrating both spatial and frequency information. The authors address limitations of existing state-space models like Mamba in handling image data by incorporating wavelet transformations and a cross-attention fusion layer, which better captures both local details and long-range dep...

This research paper focuses on a safety evaluation of DeepSeek-R1 and DeepSeek-V3 models within Chinese language contexts, an area previously underexplored. It highlights that while DeepSeek models possess strong reasoning capabilities, previous studies, primarily in English, have revealed significant safety flaws. To address the gap in Chinese safety assessments, the authors introduce CHiSafetyBe...

This paper introduces a novel deep unsupervised learning algorithm that leverages non-equilibrium thermodynamics to model complex datasets. The core idea involves a forward diffusion process that systematically degrades data structure, and then learning a reverse diffusion process to reconstruct it, creating a flexible generative model. This approach allows for efficient learning, sampling, and pr...

This document explores the Mamba architecture, a novel approach to sequence modeling that offers an efficient alternative to Transformers. It primarily investigates the role of "input selectivity" within Mamba's core component, the S6 layer, and its impact on the model's capabilities. The research proves Mamba's superiority over its predecessor, S4D, in approximating disco...

This paper introduces DeepSeek-R1, a new suite of large language models developed by DeepSeek-AI, focusing on enhancing reasoning capabilities through reinforcement learning (RL). It details the development of DeepSeek-R1-Zero, a model trained purely with RL that demonstrates strong reasoning but has readability issues, and DeepSeek-R1, which addresses these flaws by incorporating multi-stage trai...

This paper introduces DeepSeek-V3, a large Mixture-of-Experts (MoE) model designed to advance open-source language model capabilities with improved training efficiency and performance. The document details its innovative architecture, including an auxiliary-loss-free load balancing strategy and a Multi-Token Prediction objective for enhanced data efficiency and future token prediction. It further...

The provided text introduces DeepSeekMoE, an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in large language models. The authors propose two key strategies: fine-grained expert segmentation, which divides experts into smaller, more numerous units for flexible combinations, and shared expert isolation, which designates specific experts for common knowled...

This reviews a document dated January 27, 2025, from Daniel and Michael at Unsloth, details their work on quantizing DeepSeek-R1's 671B parameter model, significantly reducing its size by 80% to 131GB while maintaining functionality. They achieved this dynamic quantization by selectively applying higher bitrates to crucial layers and lower bitrates to less sensitive MoE layers, contrasting wi...

Five different sources are reviewed to understand Concept Drift in neural networks. 1) https://www.nature.com/articles/s41467-024-46142-w - Empirical data drift detection experiments on real-world medical imaging data 2) https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2024.1330258/full - One or two things we know about concept drift—a survey on monitoring in evol...

We reviewed two papers on ColBERT. We review and expand upon ColBERT, a neural information retrieval model that utilizes contextualized late interaction over BERT to estimate relevance between queries and documents. The first source, the original ColBERT paper, details its architecture, which encodes queries and documents into multi-vector representations and computes relevance by summing the maxi...

This source introduces CODEGEN, a family of large language models developed by Salesforce Research, designed for program synthesis. The models, varying in size up to 16.1B parameters, are trained on extensive natural language and programming language datasets, and the training library, JAXFORMER, is open-sourced to promote accessibility. A key contribution is the exploration of multi-turn program...

This reviews two papers on Chain of Thought: 1) https://arxiv.org/pdf/2201.11903 - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models 2) https://arxiv.org/pdf/2503.11926 - Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation These papers introduce Chain-of-Thought (CoT) prompting as a method to improve the reasoning abilities of large language model...

The provided text, titled "AI and Memory Wall," examines the growing disparity between computational power and memory bandwidth in AI, particularly for Large Language Models (LLMs). It highlights how server hardware FLOPS (floating-point operations per second) have dramatically outpaced DRAM (Dynamic Random-Access Memory) and interconnect bandwidth growth over the past two decades, leadi...

The paper "Decoupled Weight Decay Regularization" by Ilya Loshchilov and Frank Hutter (2017) introduced what we know in Pytorch now as AdamW. This academic paper explores the differences between L2 regularization and weight decay regularization in optimizing neural networks, particularly focusing on adaptive gradient algorithms like Adam. The authors demonstrate that while these two regu...

This reviews the 2015 paper which introduced Adam, an algorithm for first-order gradient-based optimization in scenarios involving stochastic objective functions. Adam uniquely computes adaptive learning rates for different parameters by estimating the first and second moments of gradients, offering advantages in computational efficiency and memory requirements. The paper details Adam's algor...

This research paper explores the scaling behavior of Transformer architectures, offering insights into pre-training and fine-tuning efficiency. It challenges previous findings by demonstrating that model shape, not just size, significantly impacts downstream task performance, unlike its lesser effect on upstream pre-training loss. The study also reveals that scaling protocols vary in effectiveness...

This 2000 paper, titled "Scaling Laws for Neural Language Models," explores the empirical relationships between the performance of neural language models (specifically Transformers) and various scaling factors: model size (parameters), dataset size (tokens), and computational budget (compute used for training). The authors demonstrate that model performance follows predictable power-law...

What ResNet introduced is adding the input of a block directly to its output, like this: Output = 𝐹(𝑥)+ 𝑥 This academic paper introduces Deep Residual Learning, a novel framework designed to facilitate the training of exceptionally deep neural networks for image recognition. The core innovation lies in reformulating layers to learn residual functions, meaning they learn the difference from the inp...

This paper introduces RoFormer, an enhanced Transformer model that leverages Rotary Position Embedding (RoPE) to improve natural language processing tasks. The authors explore existing methods for incorporating positional information into Transformer architectures, contrasting traditional additive position encoding with their novel multiplicative approach. RoPE encodes absolute position through a...

This 2000 paper introduces a novel solution to a weakness found in Long Short-Term Memory (LSTM) networks, specifically when processing continuous data streams without predefined segmentation. The core problem addressed is the unbounded growth of internal cell states within standard LSTM networks, which can lead to performance degradation. The authors propose and implement "forget gates"...

This paper introduces Longformer, a novel Transformer-based model designed to overcome the limitations of traditional Transformers in processing exceptionally long sequences. Unlike prior models with quadratic scaling, Longformer employs an attention mechanism that scales linearly with sequence length, making it efficient for documents containing thousands of tokens. This innovative architecture c...

This academic paper introduces Linformer, a novel approach to address the computational bottleneck of Transformer models in natural language processing. The authors demonstrate that the self-attention mechanism, which is a core component of Transformers and typically incurs quadratic time and space complexity with respect to sequence length, can be approximated by a low-rank matrix. By exploiting...

This paper outlines the advancements in Optical Character Recognition (OCR), particularly focusing on handwritten character and word recognition using Neural Networks. The authors, affiliated with AT&T Labs-Research, detail various machine learning techniques, including Gradient-Based Learning and Convolutional Neural Networks (CNNs) like LeNet-5, highlighting their effectiveness in handling h...