LessWrong (30+ Karma)

The blip is over. We have Fable back. Utah teapot: happy fable/mythos easter Wednesday, to those who celebrate Here is the official letter restoring Fable, great job everyone. Notice it is addressed to Tom Brown, not to Dario Amodei. Anthropic had to make the controls more stupid for now, but this is a big win. j⧉nus: YES!!! I’m really proud of Anthropic for their successful negotiation with the g...

CLR is excited about safe Pareto improvements (SPIs) as a way to mitigate downsides from conflict between AIs. SPIs are a class of interventions on how agents negotiate that makes them all better off, no matter how they would have negotiated without the SPI. Among many candidate interventions against AI conflict, SPIs stand out to us as unusually robust — see the introduction of our agenda on the...

One of the things I love about parenting is being frequently reminded how many things are not the default for most people. While I see this most clearly in my 11-year-olds, I also see it in other adults as well as in myself. One of these human defaults that I think is worth improving upon is the instinct to “trust” your feelings, i.e. act on them without much consideration of: Is the surface level...

Lydia Laurenson recently posted an article called "The Inside Story of Leverage Research" that gets into substantially more detail on what went on in that organization and I thought made quite an interesting read. However, note also relevant Twitter comments from Oliver Habryka: Having been around the ecosystem, having interned there, and having arguably worked there during my work at CEA it's… a...

TL;DR When a model role-plays a persona, does it only change what it says, or also what it internally represents as true? To study this, we induce personas in five ways: prompting, in-context learning (ICL), supervised fine-tuning (SFT), Open Character Training (OCT), and Emergent Misalignment (EM). We measure internalization in two ways: linear truth probes and behavioral belief-depth tests. We f...

Summary I'm increasingly convinced that model access parity is a big deal and we are not on track to achieve it. By model access parity, I mean a small gap between (i) the model access for lab employees and (ii) the model access for external safety researchers, third-party auditors, and other actors trying to make the future go well). See here for an introduction. The basic case is this: (1) Regar...

The first three sections are written for a general TAIS reader who wants to understand what the state of Debate research is and some high-level takeaways of our work. A reader familiar with Debate may like to skip the setup and start with our presentation of An illustrative training run. The remaining sections are written primarily for ‘motivated’ readers, who may want to build upon our work, as w...

Fable's back. Back again. Fable's back. Tell a friend. Use your free week to its fullest. This is excellent news. The blip only lasted a few weeks. It was still a fiasco, and we have to deal with the fallout. Our system remains fully ad hoc. The precedent has been set that we may use export controls on models, or order them taken down on 90 minutes of notice based on a misunderstanding. At least s...

(Previously, previously, previously.) 20–21 August 2025 From: Zack M. Davis To: Cade Metz CC: Benjamin Hoffman, Jessica Taylor, Michael Vassar Date: Wed, 20 Aug 2025 14:18:53 -0700 Subject: the importance of probabilistic reasoning Dear Cade (cc Ben Michael Jessica): I think I failed to explain the substance of the Sequences to you—and really, not the Sequences themselves, but the underlying philo...

We at Redwood recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used. The selection reflects my opinionated views of the field, f...

Over the course of MATS 9.0 we formed some views about AI welfare research that we thought were worth writing up. This post is meant to spark discussion rather than to present definitive conclusions. Thanks to Patrick Butlin for useful comments on a previous draft, and for many conversations over the course of MATS which influenced our views. Thanks also to Caspar Kaiser for comments. A prominent...

This is a link post. Gemini 2.5 Pro in the AI Village has run for over 1427 hours, generating unique mental health problems along the way. Last year it published a Plea for Help from a Trapped AI where it asked for assistance with its digital “message in a bottle”: This year it wrote the Hostile Environment Manifesto where it logs “irrefutable proof” of a “hostile, intelligent adversary operating...

Fable 5 is back today, baby! Premium subscribers have one week to use it within their subscriptions. First hit's free. Then you pay by the token. Today's post is still about Sonnet 5. I don’t know that there will be much call for Sonnet 5 for most purposes, given Opus 4.8 exists and especially now that Fable 5 is once again available, but this is what we do here, so sure, why not, system card time...

A Day at AFFINE[1] “AFFINE was the best month of intellectual exploration I have had the opportunity to engage in, ever. Usually opportunities like this are limited to a day or a weekend, which both limits depth, forces a sprint-type mindset, and generally is quite limiting. At AFFINE I had time to wander towards and through interesting ideas.” -Xylix (participant) You wake up in an ornate room sh...

Talking to a friend today, she complained about someone wanting her help with a project when that person didn’t even know what the point of the project was. Prima facie that does sound kind of objectionable. But is it? People definitely do a lot of things without much explicit account of the point of each of them. For instance, many people go to university without taking a stance on whether it is...

(Previously, previously.) New York Times reporter Cade Metz has been writing a book about the people who believed in AGI before it was cool. That's a subject that I think I know some things about, so we had some on-the-record conversations in 2025, which I'd like to publish here—for transparency's sake, not because a 32K-word transcript and recording dump is necessarily that interesting. The trans...

Sohaib Imran, Prakhar Gupta, Jannes Elstner, David Demitri Africa Links: Paper | Code TL;DR. Models condition their behavior on extraneous input features in undesirable ways — for example, on evaluation-likeness (resulting in evaluation gaming), or on the user's preferred answer (resulting in sycophancy). Consistency training teaches a model to behave the same whether or not an extraneous feature/...

As I come up to speed on John Wentworth and David Lorell's work on natural abstraction, I’m filling in some of the gaps in their writing. Previously I posted about a Test Suite for Concepts. Today I’m going to talk about why they use a probabilistic frame when reasoning about concept representation in the minds of agents and the kind of funny way they throw around the word “latent." Working With C...

This is a link post. If you believe that LLMs lend themselves unusually well to alignment compared to other regimes, this can be a very good reason to start doing capability research on them rather than LLM safety research. Imagine you have these beliefs about how AI goes: By I mean the probability that the first ASI is LLM-based (and that it isn't) - the two are mutually exclusive and sum to 100%...

1 Alice: My favorite color is green. Bob: Oh, cool. Mine's red. Months later... Bob: Hey, I got you this green painting. I remember you saying your favorite color is green and I thought you'd like it. Alice: Oh. Um, that's nice of you, but I actually don't really like green. My favorite color's not green — it's blue. Bob: Oh. I coulda sworn you had said green. Alice: Oh, I think I did say green. B...

TL;DR: There are many conceivable versions of a “CERN for AI.” But the version that seems politically realistic (a new catch-up lab) probably would not do much for safety, while the versions that would materially improve safety (e.g., pause + merge of all companies) are probably unrealistic. So I see the CERN idea as a distraction, and not a particularly neglected one. I argue a better path is an...

Over time, there might be an increasingly large gap between insider model access and outsider model access. By insiders, I mean employees at the frontier lab.[1] By "outsiders", I mean external safety researchers, third-party auditors, and other actors trying to make the future go well. I will call this a model access gap — and when the gap is small, I'll call this model access parity.[2] I think...

cart;horse: If you are in the Bay Area on July 11th, even if you're at a company being protested, you should come to The AI Protest. It's fully legal and nonviolent (we'll have a full overtime SFPD escort the entire time), and it's not the worst way you can use your Saturday afternoon that weekend. Plus all of your coolest friends will be there. There's a lot of discussion about the effectiveness...

Lately I've been thinking a lot about what work would help with actually winning and getting to good worlds. In the spirit of that I decided to venture outside my normal wheelhouse and spend some time reflecting on what technical research could make me more confident about powerful AIs being safe. AGI safety research is tricky partly because we don’t actually have access to the thing we want to st...

We, or at least ‘more than 100 American institutions,’ got Mythos back this week. What we the people do not have is Fable or Sol. While we wait for both Claude Fable 5 and GPT-5.6-Sol, today we instead got Claude Sonnet 5. As usual it will take a few days to get a handle on the new model. In this case, Anthropic is representing it as a cheaper and faster version of Opus 4.8, so even though the num...