Learning GenAI via SOTA Papers

The paper " To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning " explores the computational trade-offs between two primary strategies for scaling Large Language Model (LLM) test-time compute: sequential search (explicit backtracking within a chain-of-thought) and parallel sampling (generating multiple independent solutions and using best-of-n selection) . By evaluating...

OpenAI o3 and o4-mini System Card outlines the capabilities, safety evaluations, and risk assessments for these advanced reasoning models. Here is a short summary of its key points: Core Capabilities: The o3 and o4-mini models combine state-of-the-art reasoning with full tool capabilities , such as web browsing, Python execution, and image analysis. They are trained using reinforcement learning to...

DeepSeek-R1 is a research initiative by DeepSeek-AI focused on significantly enhancing the reasoning capabilities of Large Language Models (LLMs) through Reinforcement Learning (RL). The paper details the development of two primary models and a series of smaller distilled models: DeepSeek-R1-Zero: The researchers first explored bypassing traditional supervised fine-tuning (SFT) by training a model...

The provided paper introduces the Gemini 1.5 family of multimodal models, primarily focusing on Gemini 1.5 Pro and the highly efficient, lightweight Gemini 1.5 Flash . The defining breakthrough of these models is their capacity to process, recall, and reason over an unprecedented context window of up to 10 million tokens across text, video, and audio modalities. Here is a short summary of the key...

The provided paper introduces phi-4 , a 14-billion parameter language model developed by Microsoft Research. Unlike typical language models that rely primarily on organic web data, phi-4 achieves state-of-the-art performance for its size by intensely focusing on data quality and strategically integrating synthetic data throughout its entire training process. The model's development is built up...

The " DeepSeek-V3 Technical Report " presents DeepSeek-V3, a highly efficient and powerful Mixture-of-Experts (MoE) language model with 671 billion total parameters, of which 37 billion are activated for each token. Key Highlights of DeepSeek-V3: Innovative Architecture: The model retains the Multi-head Latent Attention (MLA) and DeepSeekMoE architectures validated in DeepSeek-V2 for eff...

The paper introduces OpenAI o1 , a new AI model designed to significantly advance capabilities in complex reasoning, mathematics, coding, and science. Here is a short summary of the key highlights: Chain of Thought Reasoning: o1 is trained using a highly data-efficient reinforcement learning algorithm that teaches it to "think" before it responds. By using a "chain of thought,"...

BitNet b1.58 is a novel 1-bit Large Language Model (LLM) variant introduced by researchers at Microsoft, where every parameter (weight) is ternary, taking on values of {-1, 0, 1} . This model aims to solve the high computational and energy costs associated with deploying traditional LLMs while maintaining their capabilities. Here are the key takeaways from the paper: Uncompromised Performance: Sta...

Qwen2.5 is a comprehensive series of large language models (LLMs) designed to handle a diverse range of tasks, featuring significant enhancements over its predecessor, Qwen2. The series offers both open-weight dense models (ranging from 0.5B to 72B parameters) and proprietary Mixture-of-Experts (MoE) models (Qwen2.5-Turbo and Qwen2.5-Plus). The key advancements of the Qwen2.5 series include: Massi...

Pixtral 12B is a 12-billion-parameter multimodal language model developed by Mistral AI, designed to seamlessly understand both text and images. Released under an open-source Apache 2.0 license, the model achieves state-of-the-art performance on various multimodal benchmarks without compromising its strong natural language reasoning capabilities. Here are the key takeaways from the paper: Innovati...

The paper presents the Qwen2-VL Series , an advanced family of Large Vision-Language Models (LVLMs) developed by the Qwen Team at Alibaba Group. Available in three parameter sizes (2B, 7B/8B, and 72B), the Qwen2-VL models achieve state-of-the-art performance that rivals leading proprietary models like GPT-4o and Claude 3.5-Sonnet across a variety of multimodal benchmarks. The models achieve this p...

The provided text is a technical report introducing the Qwen2 series , the latest suite of open-weight large language and multimodal models developed by the Qwen Team at Alibaba Group. Key highlights from the report include: Model Variants: The Qwen2 release encompasses dense models with parameter sizes of 0.5 billion, 1.5 billion, 7 billion, and 72 billion, as well as a 57 billion parameter Mixtu...

The paper introduces Chameleon, a family of early-fusion, token-based mixed-modal foundation models developed by FAIR at Meta. Unlike traditional multimodal models that rely on separate, modality-specific encoders and decoders, Chameleon utilizes a unified transformer architecture that represents both images and text as discrete tokens from the very beginning. This early-fusion approach allows the...

The provided text is a research paper introducing DeepSeek-V2 , a highly capable, open-source Mixture-of-Experts (MoE) large language model developed by DeepSeek-AI. The paper details the model's architectural innovations, training process, and evaluation results, highlighting its ability to deliver top-tier performance while maintaining economical training and highly efficient inference costs...

The technical report introduces Aya 23 , a family of open-weight, multilingual instruction-tuned language models developed by Cohere For AI that support 23 languages. Building on the previous Aya 101 model, which prioritized language breadth (covering 101 languages), Aya 23 focuses instead on an experiment in "depth versus breadth". By allocating more model capacity to fewer languages in...

The paper presents Microsoft's Phi-3 family of Small Language Models (SLMs) , notably highlighting the phi-3-mini , a 3.8 billion parameter model that is compact enough to run locally on a smartphone. Despite its small size, phi-3-mini rivals the overall performance of much larger models, such as GPT-3.5 and Mixtral 8x7B, across various academic benchmarks measuring reasoning, math, and coding...

The paper presents Llama 3 , a new family of foundation language models developed by Meta, featuring models with 8B, 70B, and a flagship 405B parameters. These models natively support multilinguality, coding, reasoning, and tool usage, with the 405B model capable of processing information in a context window of up to 128K tokens. The development of Llama 3 focuses on optimizing data, scale, and co...

The provided text is a blog post introducing Command R+ , Cohere's highly scalable large language model (LLM) specifically designed for enterprise workloads. Here is a short summary of its key features: Advanced RAG Capabilities: The model features a 128k-token context window and is highly optimized for Retrieval Augmented Generation (RAG). It improves response accuracy and provides in-line ci...

Kolmogorov-Arnold Networks (KANs) are proposed as a promising and mathematically grounded alternative to standard Multi-Layer Perceptrons (MLPs). Unlike MLPs, which apply fixed activation functions on nodes (neurons), KANs place learnable activation functions on the edges (weights) of the network. In a KAN, every weight parameter is replaced by a univariate function parameterized as a spline, mean...

Jamba is a new large language model developed by AI21 Labs that introduces a novel hybrid architecture. The model interleaves traditional Transformer layers with Mamba (a state-space model) layers, and integrates a Mixture-of-Experts (MoE) module to increase capacity without proportionally increasing compute requirements. This hybrid approach addresses the fundamental limitations of pure Transform...

DBRX is a new state-of-the-art, open, general-purpose large language model (LLM) developed by Databricks. Key Highlights: Architecture and Scale: DBRX uses a fine-grained mixture-of-experts (MoE) architecture . It contains 132 billion total parameters, but only 36 billion are active for any given input. The model was pre-trained on 12 trillion tokens of carefully curated text and code, supporting...

The provided document is a model card introducing Anthropic's Claude 3 family of large multimodal models , which consists of three models: Opus (the most capable), Sonnet (a balance of skills and speed), and Haiku (the fastest and most affordable). Key highlights from the paper include: Core Capabilities & Vision: The Claude 3 family sets new industry benchmarks in reasoning, mathematics,...

The provided paper introduces Gemma , a family of lightweight, open-weights language models developed by Google DeepMind. Built upon the foundational research, architecture, and training methodologies of Google's Gemini models, Gemma is available in two sizes: 2 billion and 7 billion parameters . Key highlights of the paper include: State-of-the-Art Performance: Gemma models outperform similar...

The paper introduces OLMo , a state-of-the-art, truly open language model designed to accelerate the scientific study of large language models. While the commercial value of language models has led to the most powerful models being closed off or only partially released (e.g., releasing only weights or inference code), OLMo provides the research community with full access to its entire development...

The paper " Textbooks Are All You Need II: phi-1.5 technical report " investigates the capabilities of small language models (SLMs) when trained on highly curated datasets. Here is a short summary of the key findings: Textbook-Quality Data : The researchers developed phi-1.5 , a 1.3 billion parameter model, by training it primarily on synthetically generated "textbook-like" dat...