Interconnects

Full post: https://www.interconnects.ai/p/grok-3-and-an-accelerating-ai-roadmap xAI launched their latest flagship model, Grok 3, last night via a live stream on X , which is a new take on the launch process, but it largely felt familiar. Grok 3 is a state-of-the-art model on some important benchmarks. The core is that it is state-of-the-art relative to available models and we know better models a...

The era we are living through in language modeling research is one characterized by complete faith that reasoning and new reinforcement learning (RL) training methods will work. This is well-founded. A day | cannot | go | by | without | a new | reasoning model , RL training result , or dataset distilled from DeepSeek R1 . The difference, compared to the last time RL was at the forefront of the AI...

Article: https://www.interconnects.ai/p/deep-research-information-vs-insight-in-science (sorry about some more audible breaths in this -- I'm going to work on it!) We at Ai2 released a local LM iPhone app for our OLMoE model (1B active, 7B total params), with greatly improved scores ! Let us know what you think, or read more here . OpenAI’s Deep Research has largely been accepted as a super valuab...

As many of you know, this weekend I appeared on the Lex Fridman Podcast with my friend Dylan Patel of SemiAnalysis to cover DeepSeek and the implications on the AI ecosystem. I recommend you check it out. This post was tricky to pull together. I decided to share it anyways given the timeliness of the topic and other more exciting things I have to get to. The minor, thematic contradictions on motiv...

This post is early to accommodate some last minute travel on my end! The new models trained to express extended chain of thought are going to generalize outside of their breakthrough domains of code and math. The “reasoning” process of language models that we use today is chain of thought reasoning. We ask the model to work step by step because it helps it manage complexity, especially in domains...

We're here to share the story of building our Open Language Models (OLMos) and what we improved to build the OLMo 2 7B/13B model that is competitive with the Llama 3.1 8B model. This is all about building an effective, small language modeling team that can share all it learns with the scientific community. Dirk, Luca, and Kyle are some of the people I learn the most from and have more knowledge (a...

Full post for links, images, etc: https://www.interconnects.ai/p/deepseek-r1-recipe-for-o1 I have a few shows to share with you this week: * On The Retort a week or two ago, we discussed the nature of AI and if it is a science (in the Kuhn’ian sense) * I appeared on Dean W. Ball and Timothy B. Lee ’s new podcast AI Summer to discuss “thinking models” and the border between post-training and reason...

Full post for images, etc: https://www.interconnects.ai/p/to-meta-ray-ban-local-ai With the Rabbit r1, the Humane pin, the Friend thing, the Sam Altman rumors, Meta Ray-Bans, and everything in between , it is obvious that we are going to get new devices in the near future driven by advancements in AI. Trying some of those that already are public makes this obvious from a functional perspective rat...

Original post: https://www.interconnects.ai/p/deepseek-v3-and-the-actual-cost-of Chapters 00:00 Opening 03:15 DeepSeek’s learning efficiency 06:49 DeepSeek’s compute transparency and reality Figures Fig 1: Benchmark Results Fig 2: ChatBotArena Results Fig 3: Compute Usage Table This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www...

Slides for this post-training talk and slides for the full tutorial on language modeling (with a bit less post-training content and no recording yet). Here are some timestamps for the video: 00:00 Introduction 10:00 Prompts & Skill Selection 14:19 Instruction Finetuning 21:45 Preference Finetuning 36:17 Reinforcement Finetuning 45:28 Open Questions 52:02 Wrap Up Psssst… we just recently released o...

In 2025 we need to disambiguate three intertwined topics: post-training, reasoning, and inference-time compute. Post-training is going to quickly become muddied with the new Reasoning Language Models (RLMs — is that a good name), given that loss functions that we studied via advancements in post-training are now being leveraged at a large scale to create new types of models. I would not call the r...

Original post https://www.interconnects.ai/p/2024-interconnects-year-in-review This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Original post: https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai Chapters 00:00 Introduction 02:51 o3 overview 05:57 Solving the Abstraction and Reasoning Corpus (ARC) 10:41 o3’s architecture, cost, and training (hint: still no tree search) 16:36 2024: RL returns Figures Fig 1, Frontier Math results Fig 2, Coding results Fig 3, ARC AGI results Fig 4, ARC AGI result details Fig 5, ARC...

Original post: https://www.interconnects.ai/p/the-ai-agent-spectrum Chapters 00:00 Introduction 03:24 Agent cartography 08:02 Questions for the near future Figures Fig 1. multiple feedbacks diagram This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Original post: https://www.interconnects.ai/p/openais-reinforcement-finetuning Chapters 00:00 Introduction 04:19 The impact of reinforcement finetuning’s existence 07:29 Hypotheses on reinforcement finetuning’s implementation Figures Fig. 1, Yann’s Cake Fig. 2, Grader config Fig. 3, RLVR learning curves This is a public episode. If you'd like to discuss this with other subscribers or get access to...

Finbarr Timbers is an AI researcher who writes Artificial Fintelligence — one of the technical AI blog’s I’ve been recommending for a long time — and has a variety of experiences at top AI labs including DeepMind and Midjourney. The goal of this interview was to do a few things: * Revisit what reinforcement learning (RL) actually is, its origins, and its motivations. * Contextualize the major brea...

Original post: https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop Figures Figure 0: OpenAI’s seminal test-time compute plot Figure 1: Setup for bucketed evals Figure 2: Evals with correctness labels Figure 3: Grouped evals Figure 4: Hypothetical inference scaling law This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visi...

Full post: https://www.interconnects.ai/p/olmo-2-and-building-language-model-training OLMo 2 demo: https://playground.allenai.org/ OLMo 2 artifacts: https://huggingface.co/collections/allenai/olmo-2-674117b93ab84e98afc72edc Chapters 00:00 Building AI Teams 06:35 OLMo 2 Figures Fig 1, pretrain plot: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmo2/pretrain.webp F...

Original post: https://www.interconnects.ai/p/tulu-3 Chapters 00:00 History 05:44 Technical details sneak peak Figures Fig 1, results: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/tulu3-img/results.webp Fig 2, overview: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/tulu3-img/overview.webp Fig 3, preferences: https://huggingface.co/...

Original post: https://www.interconnects.ai/p/scaling-realities This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Original post: https://www.interconnects.ai/p/saving-the-nairr Chapters 05:26: Do we need an AI research resource or an LM research resource? 08:59: Policy roundups This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Tim Dettmers does not need an introduction for most people building open-source AI. If you are part of that minority, you’re in for a treat. Tim is the lead developer behind most of the open-source tools for quantization: QLoRA , bitsandbytes , 4 and 8 bit inference , and plenty more. He recently finished his Ph. D. at the University of Washington, is now a researcher at the Allen Institute for AI...

Andrew Carr is co-founder and chief scientist at Cartwheel , where he is building text-to-motion AI models and products for gaming, film, and other creative endeavors. We discuss how to keep generative AI fun and expansive — niche powerful use-cases, AI poetry, AI devices like Meta RayBans, generalization to new domains like robotics, and building successful AI research cultures. Andrew is one of...

Full post: https://www.interconnects.ai/p/why-i-build-open-language-models This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

How Claude's computer use works. Where OpenAI, Anthropic, and Google all have a lead on eachother. Original post: https://www.interconnects.ai/p/claudes-agency Chapters 00:00 Claude's agentic future and the current state of the frontier models 04:43 The state of the frontier models 04:49 1. Anthropic has the best model we are accustomed to using 05:27 Google has the best small & cheap model for bu...