Large Language Model (LLM) Talk

Context engineering is the system-level discipline of architecting the dynamic information environment for AI models. Unlike prompt engineering, which focuses on phrasing specific instructions, context engineering programmatically assembles the model's "working memory" using retrieved data, tool outputs, and conversation history. It employs strategies like selection, compression, and...

Manus AI is a general-purpose autonomous agent designed to function as a digital worker rather than a passive chatbot. Developed by Monica and acquired by Meta, it utilizes a Planner-Executor architecture to orchestrate foundation models like Claude and Qwen within cloud-based sandboxes. Manus excels at complex, asynchronous tasks—including app deployment, massive parallel research, and data analy...

Kimi K2, developed by Moonshot AI, is an open agentic intelligence model built on a Mixture-of-Experts (MoE) architecture . It features 1 trillion total parameters, with 32 billion active during inference. Trained on 15.5 trillion tokens using the stable MuonClip optimizer, Kimi K2 is optimized for advanced reasoning, coding, and tool use . It offers strong performance and significantly lower pric...

Mixture-of-Recursions (MoR) is a unified framework built on a Recursive Transformer architecture, designed to enhance the efficiency of large language models. It achieves this by combining three core paradigms : parameter sharing (reusing shared layers across recursion steps), adaptive computation (dynamically assigning different processing depths to individual tokens via lightweight routers), and...

MeanFlow models introduce the concept of average velocity to fundamentally reformulate one-step generative modeling. Unlike Flow Matching, which focuses on instantaneous velocity, MeanFlow directly models the displacement over a time interval. This approach allows for highly efficient one-step or few-step generation using a single network evaluation. MeanFlow is built on a principled mathematical...

Mamba is a novel deep learning architecture that achieves linear scaling in computation and memory with sequence length, addressing Transformers' quadratic limitations. Its selective State Space Model (SSM) layer dynamically adapts to input context, allowing it to "forget" or "remember" information. Optimizations include a hardware-aware parallel algorithm for its recurrent...

LLM alignment is the process of steering Large Language Models to operate in a manner consistent with intended human goals, preferences, and ethical principles. Its primary objective is to make LLMs helpful, honest, and harmless, ensuring their outputs align with specific values and are advantageous to users. This critical process prevents unintended or harmful outputs, mitigates issues like speci...

The "Why We Think" from Lilian Weng, examines improving language models by allocating more computation at test time, drawing an analogy to human "slow thinking" or System 2. By treating computation as a resource, the aim is to design systems that can utilize this test-time effort effectively for better performance. Key approaches involve generating intermediate steps like Chain...

Deep Research is an autonomous research agent built into ChatGPT. It performs multi-step online research over several minutes, behaving like a human researcher by searching, reading, analyzing, and synthesizing information from multiple sources. It produces detailed, cited reports . Unlike standard ChatGPT's single-step responses, Deep Research uses an agent architecture orchestrating speciali...

vLLM is a high-throughput serving system for large language models. It addresses inefficient KV cache memory management in existing systems caused by fragmentation and lack of sharing, which limits batch size. vLLM uses PagedAttention, inspired by OS paging, to manage KV cache in non-contiguous blocks. This minimizes memory waste and enables flexible sharing, allowing vLLM to batch significantly m...

Qwen3 models introduce both Mixture-of-Experts (MoE) and dense architectures. They utilize hybrid thinking modes , allowing users to balance response speed and reasoning depth for tasks, controllable via parameters or tags. Developed through a multi-stage post-training pipeline, Qwen3 is trained on a significantly expanded dataset of approximately 36 trillion tokens across 119 languages. This enha...

RAGEN is a modular system for training and evaluating LLM agents using multi-turn reinforcement learning. Built on the StarPO framework , it implements the full training loop including rollout generation, reward assignment, and trajectory optimization. RAGEN serves as research infrastructure to analyze LLM agent training dynamics, focusing on challenges like stability, generalization, and the emer...

DeepSeek-Prover-V2 is an open-source large language model designed for formal theorem proving in Lean 4 . Its training relies heavily on synthetic data , generated by using DeepSeek-V3 to decompose problems into subgoals , which are then recursively solved by a smaller 7B prover model . The model uses a two-stage training process, including supervised fine-tuning and reinforcement learning (GRPO)...

The DeepSeek-Prover project aims to advance large language model capabilities in formal theorem proving by addressing the scarcity of training data. It uses autoformalization to convert informal high school and undergraduate math competition problems into formal statements, generating a large dataset of 8 million synthetic proofs. Quality filtering and formal verification with Lean 4 ensure data r...

The Model Context Protocol (MCP) , introduced by Anthropic in November 2024, is an open protocol standardizing how applications provide context to LLMs . Acting like a " USB-C port for AI applications ," it provides a standardized way to connect AI models to different data sources and tools. MCP employs a client-server architecture to overcome the complex "MxN integration problem&qu...

LLM post-training is crucial for refining the reasoning abilities developed during pretraining. It employs fine-tuning on specific reasoning tasks, reinforcement learning to reward logical steps and coherent thought processes, and test-time scaling to enhance reasoning during inference. Techniques like Chain-of-Thought (CoT) and Tree-of-Thoughts (ToT) prompting, along with methods like Monte Carlo...

Agent AI refers to interactive systems that perceive visual, language, and environmental data to produce meaningful embodied actions in physical and virtual worlds. It aims to create sophisticated and context-aware AI, potentially paving the way for AGI by leveraging generative AI and cross-reality training. Agent AI systems often use large foundation models (LLMs and VLMs) for enhanced perception...

FlashAttention-3 accelerates attention on NVIDIA Hopper GPUs through three key innovations. It achieves producer-consumer asynchrony by dividing warps into producer (data loading with TMA) and consumer (computation with asynchronous Tensor Cores) roles, overlapping these critical phases. Second, it hides softmax latency by interleaving softmax operations with asynchronous GEMMs using techniques li...

FlashAttention-2 builds upon FlashAttention to achieve faster attention computation with better GPU resource utilization . It enhances parallelism by also parallelizing along the sequence length dimension, optimizing work partitioning between thread blocks and warps to reduce shared memory access. A key improvement is the reduction of non-matmul FLOPs, which are less efficient on modern GPUs optim...

FlashAttention is an IO-aware attention mechanism designed to be fast and memory-efficient , especially for long sequences. Its core innovation is tiling , where input sequences are divided into blocks processed within the fast on-chip SRAM, significantly reducing reads and writes to the slower HBM . This contrasts with standard attention, which materializes the entire attention matrix in HBM. By...

PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that balances simplicity, stability, sample efficiency, general applicability, and strong performance. PPO replaced TRPO (Trust Region Policy Optimization) as the default algorithm at OpenAI due to its simpler implementation and greater computational efficiency, while maintaining comparable performance. PPO approximates TRPO...

Andrej Karpathy's tech talk ( youtube ), provides a comprehensive yet accessible overview of Large Language Models (LLMs) like ChatGPT. The talk details the process of building an LLM, including pre-training, data processing, and neural network training. Key stages include downloading and filtering internet text, tokenizing the text, and training neural networks to model token relationships. T...

Andrej Karpathy's talk, "Intro to Large Language Models," demystifies LLMs by portraying them as systems with two key components: a parameters file (the weights of the neural network) and a run file (the code that runs the network) . The creation of these files starts with a computationally intensive training process, where a large amount of internet text is compressed into the model's parameters....

DeepSeek-V2 is a Mixture-of-Experts (MoE) language model that balances strong performance with economical training and efficient inference. It uses a total of 236B parameters, with 21B activated for each token, and supports a context length of 128K tokens. Key architectural innovations include Multi-Head Latent Attention (MLA) , which compresses the KV cache for faster inference, and DeepSeekMoE ,...

Matrix calculus is essential for understanding and implementing deep learning. It provides the mathematical tools to optimize neural networks using gradient descent. The Jacobian matrix, a key concept, organizes partial derivatives of vector-valued functions. The vector chain rule simplifies derivative calculations in nested functions, common in neural networks. Automatic differentiation, used in...