Interconnects

Arvind Narayanan is a leading voice disambiguating what AI does and does not do. His work, with Sayash Kapoor at AI Snake Oil , is one of the few beacons of reasons in a AI media ecosystem with quite a few bad Apples. Arvind is a professor of computer science at Princeton University and the director of the Center for Information Technology Policy . You can learn more about Arvind and his work on h...

Read the full post here : https://www.interconnects.ai/p/building-on-evaluation-quicksand Chapters 00:00 Building on evaluation quicksand 01:26 The causes of closed evaluation silos 06:35 The challenge facing open evaluation tools 10:47 Frontiers in evaluation 11:32 New types of synthetic data contamination 13:57 Building harder evaluations Figures Fig 1: https://huggingface.co/datasets/natolamber...

Andrew Trask is one of the bright spots in engaging with AI policy for me in the last year. He is a passionate idealist, trying to create a future for AI that enables privacy, academic research, and government involvement in a rapidly transforming ecosystem. Trask is a leader of the OpenMined organization facilitating researcher access to non-public data and AIs, a senior research scientist at Goo...

How scaling changes model behavior Some trends are reasonable to extrapolate, some are not. Even for the trends we are succeeding at extrapolating, it is not clear how that signal translates into different AI behaviors. Read it here: https://www.interconnects.ai/p/how-scaling-changes-model-behavior [00:00] How scaling changes model behavior [05:03] Metaphors for what scaling may solve [08:45] Shor...

SB1047's veto, OpenAI's turnover, and a constant treadmill pushing AI startups to be all too similar to big technology name brands. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/ai-safety-culture-vs-capitalism 00:00 AI Safety's Crux: Culture v Capitalism 06:03 SB1047 as a regulatory l...

Riley Goodside is a staff prompting engineer at Scale AI. Previously working in data science, he is often seen as the default for the new role of a “prompt engineer.” He regularly posts incisive prompts that illicit notable behavior from the most popular AI models. I really resonated with this saying from Anthropic’s recent podcast on prompt engineering — “now we write essays and treat them as cod...

Sorry this one was late! Thanks for bearing with me, and keep sending feedback my way. Still a year or two away from when I have time to record these, but I would love to. Open-source tools, examples, limits, and the state of training multimodal models. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.inte...

What productionizing test-time compute shows us about the future of AI. Exploration has landed in language model training. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/reverse-engineering-openai-o1 00:00 Reverse engineering OpenAI's o1 01:52 From Q-star to Strawberry to o1 05:13 Trai...

Scale AI's future versus further scaling of language model performance. How Nvidia may take all the margins from the data market, too. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/ai-data-foundry 00:00 Futures of the data foundry business model 02:57 What it is like to work with data...

And why the concept of mandating "model spec's" could be a good start. (Oops, forgot to upload this yesterday!) This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post:  https://www.interconnects.ai/p/a-post-training-approach-to-ai-regulation 0:00 A post-training approach to AI regulation with Model Specs 1:45 Expanded roles...

Whether or not scaling works, we should spend more on inference. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/openai-strawberry-and-inference-scaling-laws 00:00 OpenAI's Strawberry, LM self-talk, inference scaling laws, and spending more on inference 01:51 OpenAI's Strawberry 04:16 S...

Ai2 released OLMoE, which is probably our "best" model yet relative to its peers, but not much has changed in the process. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/olmoe-and-building-better-llms 00:00 OLMoE and the hidden simplicity in training better foundation models 02:04 Fron...

The Open Source Initiative is working towards a definition. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/defining-open-source-ai 0:00 On the current definitions of open-source AI and the state of the data commons 3:17 Reasons to not mandate fully released data 4:24 Sufficient but not...

The latest model from one of the most popular fine-tuning labs makes us question how a model should be identified as a "frontier model." This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/nous-hermes-3 0:00 Nous Hermes 3 and exploiting underspecified evaluations 5:29 Parsing training lesso...

I had the pleasure of Talking with Ross Taylor , who has a great spectrum of unique experiences in the language modeling space — evaluation experience, Galactica lead author, Llama post training, etc. This is a really great conversation on the frontier of language model (LM) reasoning, LM deployments and demos, LM’s for science, RLHF, and other topics. I’ve been trying to get Ross to come on for a...

Apple, Meta, and Nvidia all agree -- synthetic data, iterative training, human preference labels, and lots of filtering. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/frontier-model-post-training 00:00 Llama 3.1 post-training and the new normal for RLHF 01:18 A new standard pipeline 0...

This week, I had the pleasure of chatting with Sebastian Raschka . Sebastian is doing a ton of work on the open language model ecosystem and AI research broadly. He’s been writing the great Ahead of AI newsletter (that has the biggest audience overlap with Interconnects, at 26%, so a lot of you know him) and multiple educational books , all on top of being a full time machine learning engineer at...

And how to understand Llama three point one's results. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/gpt-4o-mini-changed-chatbotarena 0:00 GPT-4o-mini changed ChatBotArena 3:23 Llama 3 in the arena 5:13 Partial solutions and next steps Fig 1: https://huggingface.co/datasets/natolamber...

Defining the future of the AI economy and regulation. Is Meta's AI play equivalent to the Unix stack for open-source software? This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/llama-405b-open-frontier-model 00:00 Llama 3.1 405b, Meta's AI strategy, and the new open frontier model ecosyst...

SB 1047, AI regulation, and unlikely allies for open models The rallying of the open-source community against CA SB 1047 can represent a turning point for AI regulation. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/sb-1047-and-open-weights 00:00 Introduction 01:53 SB 1047 and targeti...

I Switched to Claude 3.5 Speculations on the role of RLHF and why I love the model for people who pay attention. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/switched-to-claude-from-chatgpt 00:00 I Switched to Claude 3.5 03:57 Product priorities 05:15 RLHF's peak? Fig 1: https://hugg...

I’m really excited to resume the Interconnects Interviews with Dean W. Ball from the Hyperdimensional Substack (you should subscribe). We cover the whole stack of recent happenings in AI policy, focusing of course on California’s bill SB 1047. We cover many, many more great topics here including: * What will happen in the case of a minor AI disaster, * If Meta will release the 405B model, and why,...

Things to be aware of if you work on language model fine-tuning. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/rlhf-roundup-2024 00:00 RLHF Roundup: Trying to get good at PPO, charting RLHF's impact, RewardBench retrospective, and a reward model competition 04:32 How big is the impact...

Synthetic data is known to be a super powerful tool for every level of the language modeling stack. It's documented as being used for expanding vanilla pretraining data and creating large swaths of fine-tuning data. Many, many more rumors surround its use, Anthropic's pretraining-scale constitutional AI, Mistral AI's first models being pretrained on OpenAI outputs, Q-star's hopes as OpenAI's remai...

Signs point to a general-use Sora-like model coming very soon, maybe even with open-weights. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/text-to-video-ai-is-already-abundant 0:00 Text-to-video AI is already abundant 5:08 What's next for the text-to-video market? 6:49 Are text-to-vid...