Doom Debates!

My friend John from the AI X-Risk Network podcast got hired by the Center for AI Safety (CAIS) two weeks ago. Today I suddenly learned he’s been fired. I’m frustrated by this decision, and frustrated with the whole AI x-risk community’s weak messaging. Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online Hope to see you there! Watch the Lethal Intelligence Guide, the ult...

Prof. Gary Marcus is a scientist, bestselling author and entrepreneur, well known as one of the most influential voices in AI. He is Professor Emeritus of Psychology and Neuroscience at NYU.  He was founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016. Gary co-authored the 2019 book, Rebooting AI: Building Artificial Intelligence We Can Trust , and the 202...

 Dr. Mike Israetel, renowned exercise scientist and social media personality, and more recently a low-P(doom) AI futurist, graciously offered to debate me! 00:00 Introducing Mike Israetel 12:19 What’s Your P(Doom)™ 30:58 Timelines for Artificial General Intelligence 34:49 Superhuman AI Capabilities 43:26 AI Reasoning and Creativity 47:12 Evil AI Scenario 01:08:06 Will the AI Cooperate With Us? 01:...

I want to be transparent about how I’ve updated my mainline AI doom scenario in light of safe & useful LLMs. So here’s where I’m at… 00:00 Introduction 07:59 The Dangerous Threshold to Runaway Superintelligence 18:57 Superhuman Goal Optimization = Infinite Time Horizon 21:21 Goal-Completeness by Analogy to Turing-Completeness 26:53 Intellidynamics 29:13 Goal-Optimization Is Convergent 31:15 Early...

What’s the most likely (“mainline”) AI doom scenario? How does the existence of LLMs update the original Yudkowskian version? I invited my friend Jim Babcock to help me answer these questions. Jim is a member of the LessWrong engineering team and its parent organization, Lightcone Infrastructure. I’ve been a longtime fan of his thoughtful takes. This turned out to be a VERY insightful and informat...

Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common. 00:00 Introducing Ozzie 02:18 The Rationality Community 06:32 What’s Your P(Doom)™ 08:09 High-Quality Discourse and Social Media 14:17 Guesstimate and Squ...

David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment . We dive into David’s impressive career, his high P(Doom...

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it. 00:00 Overview of AI 2027 05:13 2025: Stumbling Agents 16:23 2026: Advanced Agents 21:49 2027: The Intelligence Explosion 29:13 AI's Initial Exploits and OpenBrain's Secrecy 30:41 Agent-3 and the Rise of Super...

Dr. Peter Berezin is the Chief Global Strategist and Director of Research at BCA Research, the largest Canadian investment research firm. He’s known for his macroeconomics research reports and his frequent appearances on Bloomberg and CNBC. Notably, Peter is one of the only macroeconomists in the world who’s forecasting AI doom! He recently published a research report estimating a “ more than 50/5...

Nathan Labenz, host of The Cognitive Revolution , joins me for an AI news & social media roundup! 00:00 Introducing Nate 05:18 What’s Your P(Doom)™ 23:22 GPT-4o Image Generation 40:20 Will Fiverr’s Stock Crash? 47:41 AI Unemployment 55:11 Entrepreneurship 01:00:40 OpenAI Valuation 01:09:29 Connor Leahy’s Hair 01:13:28 Mass Extinction 01:25:30 Is anyone feeling the doom vibes? 01:38:20 Rethinking A...

In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks. 00:00 Introduction 01:47 Defining Doom and AI Risks 05:53 P(Doom) 10:04 Doom Debates’ Mission 16:17 P...

Alexander Campbell claims that having superhuman intelligence doesn’t necessarily translate into having vast power, and that Gödel's Incompleteness Theorem ensures AI can’t get too powerful. I strongly disagree. Alex has a Master's of Philosophy in Economics from the University of Oxford and an MBA from the Stanford Graduate School of Business, has worked as a quant trader at Lehman Brothers and B...

Roko Mijic has been an active member of the LessWrong and AI safety community since 2008. He’s best known for “Roko’s Basilisk”, a thought experiment he posted on LessWrong that made Eliezer Yudkowsky freak out, and years later became the topic that helped Elon Musk get interested in Grimes. His view on AI doom is that:* AI alignment is an easy problem* But the chaos and fighting from building sup...

Sir Roger Penrose is a mathematician, mathematical physicist, philosopher of science, and Nobel Laureate in Physics. His famous body of work includes Penrose diagrams, twistor theory, Penrose tilings, and the incredibly bold claim that intelligence and consciousness are uncomputable physical phenomena related to quantum wave function collapse.  Dr. Penrose is such a genius that it's just interesti...

The Center for AI Safety just dropped a fascinating paper — they discovered that today’s AIs like GPT-4 and Claude have preferences! As in, coherent utility functions. We knew this was inevitable, but we didn’t know it was already happening. This episode has two parts: In Part I (48 minutes), I react to David Shapiro’s coverage of the paper and push back on many of his points. In Part II (60 minut...

My friend Gil Mark, who leads generative AI products at LinkedIn, thinks competition among superintelligent AIs will lead to a good outcome for humanity. In his view, the alignment problem becomes significantly easier if we build multiple AIs at the same time and let them compete. I completely disagree, but I hope you’ll find this to be a thought-provoking episode that sheds light on why the align...

Why does the simplest AI imaginable, when you ask it to help you push a box around a grid, suddenly want you to die? AI doomers are often misconstrued as having "no evidence" or just "anthropomorphizing". This toy model will help you understand why a drive to eliminate humans is NOT a handwavy anthropomorphic speculation, but rather something we expect by default from any sufficiently powerful sea...

Bryan Cantrill, co-founder of Oxide Computer, says in his talk that engineering in the physical world is too complex for any AI to do it better than teams of human engineers. Success isn’t about intelligence; it’s about teamwork, character and resilience. I completely disagree. 00:00 Introduction 02:03 Bryan’s Take on AI Doom 05:55 The Concept of P(Doom) 08:36 Engineering Challenges and Human Inte...

Thanks for everyone who participated in the live Q&A on Friday! The topics covered include advice for computer science students, working in AI trustworthiness, what good AI regulation looks like, the implications of the $500B Stargate project, the public's gradual understanding of AI risks, the impact of minor AI disasters, and the philosophy of consciousness. 00:00 Advice for Comp Sci Students 01...

It’s time for AI Twitter Beefs #3: 00:00 Introduction 01:27 Marc Andreessen vs. Sam Altman 09:15 Mark Zuckerberg 35:40 Martin Casado 47:26 Gary Marcus vs. Miles Brundage Bet 58:39 Scott Alexander’s AI Art Turing Test 01:11:29 Roon 01:16:35 Stephen McAleer 01:22:25 Emmett Shear 01:37:20 OpenAI’s “Safety” 01:44:09 Naval Ravikant vs. Eliezer Yudkowsky 01:56:03 Comic Relief 01:58:53 Final Thoughts Sho...

Effective Altruism has been a controversial topic on social media, so today my guest and I are going to settle the question once and for all: Is it good or bad? Jonas Sota is a Software Engineer at Rippling, BA in Philosophy from UC Berkeley, who’s been observing the Effective Altruism (EA) movement in the San Francisco Bay Area for over a decade… and he’s not a fan. 00:00 Introduction 01:22 Jonas...

Matthew Adelstein, better known as Bentham’s Bulldog on Substack, is a philosophy major at the University of Michigan and an up & coming public intellectual. He’s a rare combination: Effective Altruist, Bayesian, non-reductionist, theist. Our debate covers reductionism, evidence for god, the implications of a fine-tuned universe, moral realism, and AI doom. 00:00 Introduction 02:56 Matthew’s Resea...

Prof. Kenneth Stanley is a former Research Science Manager at OpenAI leading the Open-Endedness Team in 2020-2022. Before that, he was a Professor of Computer Ccience at the University of Central Florida and the head of Core AI Research at Uber. He coauthored Why Greatness Cannot Be Planned: The Myth of the Objective , which argues that as soon as you create an objective, then you ruin your abilit...

OpenAI just announced o3 and smashed a bunch of benchmarks (ARC-AGI, SWE-bench, FrontierMath)! A new Anthropic and Redwood Research paper says Claude is resisting its developers’ attempts to retrain its values! What’s the upshot — what does it all mean for P(doom)? 00:00 Introduction 01:45 o3’s architecture and benchmarks 06:08 “Scaling is hitting a wall” 🤡 13:41 How many new architectural insigh...

This week Liron was interview by Gaëtan Selle on @the-flares about AI doom. Cross-posted from their channel with permission. Original source: https://www.youtube.com/watch?v=e4Qi-54I9Zw 0:00:02 Guest Introduction 0:01:41 Effective Altruism and Transhumanism 0:05:38 Bayesian Epistemology and Extinction Probability 0:09:26 Defining Intelligence and Its Dangers 0:12:33 The Key Argument for AI Apocaly...