LessWrong (Curated & Popular)

I'm trying to build my own art of rationality training, and I've started talking to various CFAR instructors about their experiences – things that might be important for me to know but which hadn't been written up nicely before. This is a quick write up of a conversation with Andrew Critch about his takeaways. (I took rough notes, and then roughly cleaned them up for this. I don&apo...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/duvzdffTzL3dWJcxn/believing-in-1 Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post] ✓

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/5jdqtpT6StjKDKacw/attitudes-about-applied-rationality Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓

This is a hasty speculative fiction vignette of one way I expect we might get AGI by January 2025 (within about one year of writing this). Like similar works by others, I expect most of the guesses herein to turn out incorrect. However, this was still useful for expanding my imagination about what could happen to enable very short timelines, and I hope it's also useful to you. The assistant o...

This is a linkpost for https://garrisonlovely.substack.com/p/sam-altmans-chip-ambitions-undercut If you enjoy this, please consider subscribing to my Substack. Sam Altman has said he thinks that developing artificial general intelligence (AGI) could lead to human extinction, but OpenAI is trying to build it ASAP. Why? The common story for how AI could overpower humanity involves an “intelligence e...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/PhTBDHu9PKJFmvb4p/a-shutdown-problem-proposal Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post] ✓

People often parse information through an epistemic consensus filter. They do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post] ✓

Background Terminology: Counterfactual Impact vs “Leading The Parade” Y’know how a parade or marching band has a person who walks in front waving a fancy-looking stick up and down? Like this guy:  The classic 80's comedy Animal House features a great scene in which a prankster steals the stick, and then leads the marching band off the main road and down a dead-end alley. That is not the guy w...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/kcKrE9mzEHrdqtDpE/the-case-for-ensuring-that-powerful-ais-are-controlled Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post] ✓

I often encounter some confusion about whether the fact that synapses in the brain typically fire at frequencies of 1-100 Hz while the clock frequency of a state-of-the-art GPU is on the order of 1 GHz means that AIs think "many orders of magnitude faster" than humans. In this short post, I'll argue that this way of thinking about "cognitive speed" is quite misleading. The...

A pdf version of this report is available here. Summary. In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development. In Section 1 we discuss the tasks which are the focus of...

This is a linkpost for https://rootsofprogress.org/the-block-funding-model-for-scienceWhen Galileo wanted to study the heavens through his telescope, he got money from those legendary patrons of the Renaissance, the Medici. To win their favor, when he discovered the moons of Jupiter, he named them the Medicean Stars. Other scientists and inventors offered flashy gifts, such as Cornelis Drebbel&apo...

Let your every day be full of joy, love the child that holds your hand, let your wife delight in your embrace, for these alone are the concerns of humanity.[1] — Epic of Gilgamesh - Tablet X Say we want to train a scientist AI to help in a precise, narrow field of science (e.g. medicine design) but prevent its power from being applied anywhere else (e.g. chatting with humans, designing bio-weapons...

We are organising the 9th edition without funds. We have no personal runway left to do this again. We will not run the 10th edition without funding. In a nutshell: Last month, we put out AI Safety Camp's funding case.  A private donor then decided to donate €5K.    Five more donors offered $7K on Manifund.  For that $7K to not be wiped out and returned, another $21K in funding is needed. At t...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Crossposted from substack . As we all know, sugar is sweet and so are the $30B in yearly revenue from the artificial sweetener industry. Four billion years of evolution endowed our brains with a simple, straightforward mechanism to make sure we occasionally get an energy refuel so we can continue the fora...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/tEPHGZAb63dfq2v8n/how-useful-is-mechanistic-interpretability Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post]  ✓

This is a linkpost for https://arxiv.org/abs/2401.05566 Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated Source: https://www.lesswrong.com/posts/ZAsJv7xijKTfZkMtr/sleeper-agents-training- deceptive-llms-that-persist-through Narrated for LessWrong by Perrin Walker . Share feedback on this narration. [Curated Post] ✓ [ 125+ Karma Post] ✓

I wrote this entire post in February of 2023, during the fallout from the TIME article. I didn't post it at the time for multiple reasons: because I had no desire to get involved in all that nonsense because I was horribly burned out from my own community conflict investigation and couldn't stand the thought of engaging with people online because I generally think it's bad to post o...

"( Cross-posted from my website . Audio version here , or search "Joe Carlsmith Audio" on your podcast app.)" T his is the first essay in a series that I’m calling “Otherness and control in the age of AGI.” See here for more about the series as a whole. ) When species meet The most succinct argument for AI risk, in my opinion, is the “second species” argument. Basically, it goe...

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Following on from our recent paper, “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training”, I’m very excited to announce that I have started leading a new team at Anthropic, the Alignment Stress-Testing team, with Carson Denison and Monte MacDiarmid as current team members. Our mission—an...

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is a linkpost for https://arxiv.org/abs/2401.05566I'm not going to add a bunch of commentary here on top of what we've already put out, since we've put a lot of effort into the paper itself, and I'd mostly just recommend reading it directly, especially since there are a lot of subtle res...

Support ongoing human narrations of LessWrong's curated posts: www.patreon.com/LWCurated The goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified: Optimization. Specifically, this will be a type of Vingean agency . It will split into Selection vs Control variants. Reference (the relationship which holds between map...

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Thanks to Clément Dumas, Nikola Jurković, Nora Belrose, Arthur Conmy, and Oam Patel for feedback. In the comments of the post on Google Deepmind's CCS challenges paper, I expressed skepticism that some of the experimental results seemed possible. When addressing my concerns, Rohin Shah made some claims alon...

(Cross-posted from my website. Audio version here, or search "Joe Carlsmith Audio" on your podcast app. This is the first essay in a series that I’m calling “Otherness and control in the age of AGI.” See here for more about the series as a whole.) When species meet The most succinct argument for AI risk, in my opinion, is the “second species” argument. Basically, it goes like this. Premi...