LessWrong (Curated & Popular)

davidad has a 10-min talk out on a proposal about which he says: “the first time I’ve seen a concrete plan that might work to get human uploads before 2040, maybe even faster, given unlimited funding”. I think the talk is a good watch, but the dialogue below is pretty readable even if you haven't seen it. I'm also putting some summary notes from the talk in the Appendix of this dialogue....

I guess there’s maybe a 10-20% chance of AI causing human extinction in the coming decades, but I feel more distressed about it than even that suggests—I think because in the case where it doesn’t cause human extinction, I find it hard to imagine life not going kind of off the rails. So many things I like about the world seem likely to be over or badly disrupted with superhuman AI (writing, explai...

Recently, I have been learning about industry norms, legal discovery proceedings, and incentive structures related to companies building risky systems. I wanted to share some findings in this post because they may be important for the frontier AI community to understand well.  TL;DR Documented communications of risks (especially by employees) make companies much more likely to be held liable in co...

Support ongoing human narrations of curated posts: www.patreon.com/LWCurated This is a linkpost for https://transformer-circuits.pub/2023/monosemantic-features/ Text of post based on our blog post as a linkpost for the full paper which is considerably longer and more detailed. Neural networks are trained on data, not programmed to follow rules. We understand the math of the trained network exactly...

Support ongoing human narrations of curated posts: www.patreon.com/LWCurated (You can sign up to play deception chess here if you haven't already.) This is the first of my analyses of the deception chess games. The introduction will describe the setup of the game, and the conclusion will sum up what happened in general terms; the rest of the post will mostly be chess analysis and skippable if...

This is a linkpost for https://nitter.net/ESYudkowsky/status/1718654143110512741 Comp sci in 2017: Student:  I get the feeling the compiler is just ignoring all my comments. Teaching assistant:  You have failed to understand not just compilers but the concept of computation itself. Comp sci in 2027: Student:  I get the feeling the compiler is just ignoring all my comments. TA:  That's weird. ...

A common theme implicit in many AI risk stories has been that broader society will either fail to anticipate the risks of AI until it is too late, or do little to address those risks in a serious manner. In my opinion, there are now clear signs that this assumption is false, and that society will address AI with something approaching both the attention and diligence it deserves. For example, one c...

I examined all the biorisk-relevant citations from a policy paper arguing that we should ban powerful open source LLMs. None of them provide good evidence for the paper's conclusion. The best of the set is evidence from statements from Anthropic -- which rest upon data that no one outside of Anthropic can even see, and on Anthropic's interpretation of that data. The rest of the evidence...

This is a linkpost for https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/ Released today (10/30/23) this is crazy, perhaps the most sweeping action taken by government on AI yet.  Below, I've segmented by x-risk and non-x-risk related proposals, excluding the proposal...

Over the next two days, the UK government is hosting an AI Safety Summit focused on “the safe and responsible development of frontier AI”. They requested that seven companies (Amazon, Anthropic, DeepMind, Inflection, Meta, Microsoft, and OpenAI) “outline their AI Safety Policies across nine areas of AI Safety”. Below, I’ll give my thoughts on the nine areas the UK government described; I’ll note k...

Support ongoing human narrations of curated posts: www.patreon.com/LWCurated Previously: Sadly, FTX I doubted whether it would be a good use of time to read Michael Lewis’s new book Going Infinite about Sam Bankman-Fried (hereafter SBF or Sam). What would I learn that I did not already know? Was Michael Lewis so far in the tank of SBF that the book was filled with nonsense and not to be trusted? I...

Timaeus is a new AI safety research organization dedicated to making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. Currently, we are working on singular learning theory and developmental interpretability . Over time we expect to work on a broader research agenda, and to create understanding-based evals informed by our research. Source: http...

I am excited about AI developers implementing responsible scaling policies ; I’ve recently been spending time refining this idea and advocating for it. Most people I talk to are excited about RSPs, but there is also some uncertainty and pushback about how they relate to regulation. In this post I’ll explain my views on that: I think that sufficiently good responsible scaling policies could dramati...

AI used to be a science. In the old days (back when AI didn't work very well), people were attempting to develop a working theory of cognition. Those scientists didn’t succeed, and those days are behind us. For most people working in AI today and dividing up their work hours between tasks, gone is the ambition to understand minds. People working on mechanistic interpretability (and others att...

Some brief thoughts at a difficult time in the AI risk debate. Imagine you go back in time to the year 1999 and tell people that in 24 years time, humans will be on the verge of building weakly superhuman AI systems. I remember watching the anime short series The Animatrix at roughly this time, in particular a story called The Second Renaissance I part 2 II part 1 II part 2 . For those who haven&a...

Judea Pearl is a famous researcher, known for Bayesian networks (the standard way of representing Bayesian models), and his statistical formalization of causality. Although he has always been recommended reading here , he's less of a staple compared to, say, Jaynes. So the need to re-introduce him. My purpose here is to highlight a soothing, unexpected show of rationality on his part. One yea...

Views are my own, not Open Philanthropy’s. I am married to the President of Anthropic and have a financial interest in both Anthropic and OpenAI via my spouse. Over the last few months, I’ve spent a lot of my time trying to help out with efforts to get responsible scaling policies adopted. In that context, a number of people have said it would be helpful for me to be publicly explicit about whethe...

Support ongoing human narrations of curated posts: www.patreon.com/LWCurated Doomimir : Humanity has made no progress on the alignment problem. Not only do we have no clue how to align a powerful optimizer to our "true" values, we don't even know how to make AI "corrigible"—willing to let us correct it. Meanwhile, capabilities continue to advance by leaps and bounds. All i...

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Jeffrey Ladish.  TL;DR LoRA fine-tuning undoes the safety training of Llama 2-Chat 70B with one GPU and a budget of less than $200. The resulting models [1] maintain helpful capabilities without refusing to fulfill harmful instructions. We show that, if model weights are released, safety...

Holly is an independent AI Pause organizer, which includes organizing protests (like this upcoming one ). Rob is an AI Safety YouTuber . I (jacobjacob) brought them together for this dialogue, because I've been trying to figure out what I should think of AI safety protests, which seems like a possibly quite important intervention; and Rob and Holly seemed like they'd have thoughtful and...

Three of the big AI labs say that they care about alignment and that they think misaligned AI poses a potentially existential threat to humanity. These labs continue to try to build AGI. I think this is a very bad idea. The leaders of the big labs are clear that they do not know how to build safe, aligned AGI. The current best plan is to punt the problem to a (different) AI, and hope that can solv...

Support ongoing human narrations of curated posts: www.patreon.com/LWCurated How do you affect something far away, a lot, without anyone noticing? (Note: you can safely skip sections. It is also safe to skip the essay entirely, or to read the whole thing backwards if you like.) Source: https://www.lesswrong.com/posts/R3eDrDoX8LisKgGZe/sum-threshold-attacks Narrated for LessWrong by Perrin Walker ....

Last year, I wrote about the promise of gene drives to wipe out mosquito species and end malaria. In the time since my previous writing, gene drives have still not been used in the wild, and over 600,000 people have died of malaria. Although there are promising new developments such as malaria vaccines , there have also been some pretty bad setbacks (such as mosquitoes and parasites developing res...

COI: I am a research scientist at Anthropic, where I work on model organisms of misalignment ; I was also involved in the drafting process for Anthropic’s RSP . Prior to joining Anthropic, I was a Research Fellow at MIRI for three years. Thanks to Kate Woolverton, Carson Denison, and Nicholas Schiefer for useful feedback on this post. Recently, there’s been a lot of discussion and advocacy around...

Patreon to support human narration. (Narrations will remain freely available on this feed, but you can optionally support them if you'd like me to keep making them.) *** Epistemic status: model which I find sometimes useful, and which emphasizes some true things about many parts of the world which common alternative models overlook. Probably not correct in full generality. Consider Yoshua Ben...