LessWrong (30+ Karma)

TL;DR: Current LLMs are bad communicators relative to their agentic capabilities. I claim that articulacy is useful (and perhaps necessary) for AI safety and suggest a path for improving articulacy. Briefly: a theory for articulacy Frequently, LLM agents miscommunicate with their human operators, such as when they write documentation or respond to queries about their activity during a coding sessi...

In keeping with my tradition and given I will lose access to Fabel in a couple days, I have asked fable to read all my organically written short stories and attempt to write one in my style. It output the following: There's LLM-written text here. Claude Fable writes: en-US-AvaMultilingualNeural__ The Last Complainer I want it on the record that I complained about the job before I took it. This is,...

There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here. I encourage reading of the whole original blog post or paper, if you have the time. Table of Contents Through A Different Lens. Establishing J-Space As A Global Workspace. Are You Pondering What I’m Pondering? Assistant J. The Power Of Virtuous...

This post summarizes some of my views. It states uncertain things boldly in part to [surface disagreements / help readers point out things that seem wrong or missing]. Note that many things are uncertain and many claims are not justified here. One tl;dr: money is great even on the margin, and it's crucial to invest extremely well rather than merely well. In 2026, philanthropists will donate about...

This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase. J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback throughout. TLDR Models can acquire undesirable traits from during supervised fine-tuning (SFT). A natural thing to try is to identify the data points with...

I just had one of those delightful moments where I have a very specific idea, and then I search for it (Claude Research in this case), and it turns out that people have already been using that exact concept but just calling it by a name I'd never heard of. That name is conceptor, but the basic construction is not new and has gone by many other names. What a conceptor is I'll quote from "Conceptors...

This is a synthesis of many facts I've learned over the last few years, mainly about metabolism and exercise,[1] that have helped me become much healthier, and might help you. It's focused mainly on “basic” models that, in my opinion, high school health class ought to cover (though mine emphatically did not). Because of various personality diso... traits, simply doing what people tell me to do bas...

Exercise is hard but it's even harder if you have to use your brain and muscles at the same time. I wish a personal trainer would just teleport into my house whenever I work out, tell me exactly what to do, and then record my progress (and complaints) to improve the program going forward. Apps are too rigid or too complicated; personal trainers are expensive and require scheduling, but using Claud...

In our last post, we argued that measuring evaluation awareness is fundamentally challenging because of the safe-to-dangerous distributional shift: we cannot directly measure the evaluation awareness of a model without deploying it, but we cannot safely deploy it until we know it is not scheming. We expect sufficiently superhuman AI will be eval aware, but this post outlines a tentative solution f...

TLDR LLMs appear to have functional welfare: coherent sets of behaviour that track how well things are going relative to their goals.   Improving model functional welfare matters for safety (low welfare may amplify misalignment) and for moral reasons (models may be or become moral patients).  Naive interventions can fail in non-obvious ways. We argue any suc...

The below is a public review Anthropic asked me to write for their new global workspace paper. I recommend at least skimming their paper first. TLDR: I think this is a fantastic paper - it presents compelling evidence for some kind of "cognitive space" in models, that is used as a "working memory" for intermediate variables during a forward pass, shows that J-Lens is a useful technique for accessi...

I recently served as a recommender in SFF's annual funding round (grants will be decided and announced in September). I'm deeply grateful for SFF's funders, and I hope more AI safety donors appear in the future. Unfortunately, the SFF experience is bad for both applicants and recommenders, it's slow, and it lacks some other desiderata. (The actual funding decisions are moderately good by my lights...

When John told me (Gretta) his practice of “visioning,” I was skeptical at first. I gave it a try, a little bit out of spite, to show him I was capable of it. It blew me away. Here's what I told him about it a few days later, edited for brevity and clarity. Something internal really felt different. Awake, alive, hungry, energetic. Selfish. A little angry. Do you know the 1984 Apple Macintosh ad? I...

[This is the blog post for our new paper Verbalizable Representations Form a Global Workspace in Language Models Readers might also be interested in: the Public commentary, Github and Neuronpedia] As you read this sentence, circuits in your brain are adjusting your posture, controlling your breathing, and transforming lines and curves on the screen into recognizable words. Most of this processing...

Epistemic status: pretty confident in the validity of the core proposal, not that confident in specific implementation details TL;DR: we should cryptographically verify that sub-agent instances/sessions are downstream of human instructions Frontier AI labs have started using LLM-based monitoring systems to check for misbehavior from their internal AI agents, which often run unwatched by humans for...

This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training. TL;DR Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on the training distribution.[1] That's true even if the tr...

It was January 27, 1986, the night before the Space Shuttle Challenger was scheduled for launch. The goal was to have a shuttle that could land back on Earth and be reused for future missions; its first-planned priorities were satellite deployment, comet observation, and science education. The last of these would involve students from around the world watching live broadcasts from Christa McAuliff...

Training-run assessments conducted by a 3rd party should become a standard part of frontier AI safety. By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer...

Ron's face pulled into a scowl. "If you don't like Quidditch, you don't have to make fun of it!" "If you can't criticise, you can't optimise. I'm suggesting how to improve the game. And it's very simple. Get rid of the Snitch." "They won't change the game just 'cause you say so!" "I am the Boy-Who-Lived, you know. People will listen to me. And maybe if I can persuade them to change the game at Hog...

Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Maria Kostylew for helpful draft feedback. Introduction This post advocates a perspective of LLMs as seeking to minimise prediction error with respect to their world models. We can moreover interpret token outputs and their scaffolded consequences as actions that close a control loop between AIs' predi...

I ran a small single-blind ZBiotics RCT at a party I hosted recently. I prepared 30 cups, 21 of which contained a placebo and 9 of which contained a shot of ZBiotics Pre-Alcohol. People took these as they entered the party. The day after, 28[1] of them filled out a form asking how hungover they felt using a standard Acute Hangover Scale (AHS).[2] At N=28 and only 9 people receiving the real treatm...

Summary The primary ToC for control makes the case that control is compelling even if it does not scale to ASI. I think it is underdiscussed that this is also true for alignment (for all the same reasons). Even though control does not need to scale to ASI, the further control does scale, the better. This is also true of alignment, and alignment seems more likely to scale further. This is my m...

I have sometimes been asked with respect to my fiction, "What the hell is a 'gendertrope'?" "Gendertrope" is a word from the language of the fictional world of dath ilan; it appears in my stories about dath ilan and dath ilani. I have observed that (1) the meaning of "gendertrope" seems hard for many people to grasp initially, and (2) once people do grasp that meaning, they sometimes use the word...

In a previous post, I explain why the universe is probably not stable, but nevertheless unlikely to be intentionally destroyable even in the limit of advanced technology. Now let's turn our attention to more prosaic risks where exotic physics merely destroys the Solar System, Earth, or just outperforms traditional nuclear weapons on some more local scale. The basic logic behind any bomb is a self-...

Decision theory is back in fashion (defining fashion as "one good post on a good EA blog"). Bentham's Bulldog (BB) has published a case against FDT (functional decision theory), contrasting rationalist enthusiasm with academic scepticism: "Academic decision theorists don't like the theory. The number of academic decision theorists who adopt it could be counted on one hand by someone missing four o...