LessWrong (30+ Karma)

In Parts 1, 2, and 3 we estimated how fast a post-AGI economy could grow using existing or historically observed production techniques, grounded in US input-output data. That approach gave us confidence that the methods we assumed were physically realizable, because they correspond to manufacturing processes that have run at scale in the past or run today. Now I would like to relax that constraint...

Summary We've been building a theory of how prompt injections work under the hood. We show it comes down to how LLMs perceive roles (the humble chat template tags). We use this theory to create new attacks, explain some weird mech interp results, and predict when attacks work. We also advocate for a new subfield focused on the science of roles, and sketch some unexplored new research problems. Wor...

I've been playing Coup for a long time now. I keep a copy in my backpack and bring it everywhere, and it's earned the space. A few reasons it's so good: It's trivial to teach. You can explain the rules in a minute or two. Anyone can pick it up and start playing immediately. Many people find it fun. Almost everyone I've played with has loved it — pretty much unanimously, not least because...

Here's Holden Karnofsky: I tend to think it's worse than 51/49. I tend to think we’re always going to be prone to overestimate how robustly good our actions are. And the more we learn about all the galaxy-brained considerations that one should have had in one's head, the more it's going to be like 50+ε%. I think AI safety is a great cause to work in. I’m excited to work in it. I think it's high im...

Natural language autoencoders are a really cool mostly-unsupervised method for producing free-form text explanations of LLM activations. You should read that paper (or the blog post) about them before reading this. I trained[1] several Qwen3-8B NLAs with different length penalties: during RL, I subtracted the token count multiplied by the length penalty hyperparameter (λ) from the RL reward[2]. I...

Paper here, code, benchmark. Builds on the preview we posted in January. Authors: @monika_j , @ma-martinez , @ollie, @Tyler Tracy We are releasing MonitoringBench, a difficulty-graded benchmark of 2,644 successful attack trajectories for evaluating coding-agent monitors, alongside the semi-automated red-teaming pipeline we used to generate it. The pipeline decomposes attack construction into strat...

Only three days after the release of Claude Fable 5, Anthropic was forced by the United States Government to make it unavailable, when a jailbreak was brought to its attention, rather than the previous situation of ‘yes obviously experts can jailbreak anything if they care enough’ and ‘yes obviously you can ask Fable to fix your code.’ Three days was enough time for many of us to learn to love Fab...

Tldr: Most strategic writing on AI governance on LessWrong describes the outsider game, which is most often visible: press, statements, open letters. Here I want to describe the other, invisible half: the insider work within ministerial cabinets and international fora, and the work of people within national and international institutions. Here are a few claims that I defend in the post: A huge par...

Daniel Drucker pointed me at a fun bug in Google's calculator: the parsec is wrong when you do math on it. As the earth travels around the sun, closer stars appear to shift back and forth against the far-distant background stars. The closer the star is the bigger this effect is. Think of how when you switch which eye you're looking through you notice near things shifting relative to farther ones....

Work also done with Cindy Wu, Asic Q Chen, Jean Tarbouriech, Min Ma, Brendan O'Donoghue, João Gabriel Lopes de Oliveira. Paper here: https://arxiv.org/abs/2606.20560 Overview In a recent collaboration between the GDM interpretability team and the GDM text diffusion team, we performed a transparency audit of DiffusionGemma, GDM's new text diffusion model. Overall, we find that DiffusionGemma is not...

I recently made a dumb (in retrospect) mistake that set me back a lot. Feeling upset and regretful, I spoke to an older family member who reassured me, "yeah, unfortunately there's no way around it; we have to experience these mistakes personally in order to learn from them". I thought, is that actually true? Can't we learn from other people's mistakes? After all, isn't that the whole point of stu...

If the box contains a diamond, I desire to believe that the box contains a diamond; If the box does not contain a diamond right now, but will contain a diamond if I believe there is a diamond, Uh... Holding unfounded beliefs might sometimes, by some cruel irony, produce better outcomes than being rational. (This post was inspired by a couple cases where the causal effect of belief seems hand-waved...

These are some personal notes taken and later dressed up a bit to make into a post. Dunno how much value is here for people already familiar with the AI Safety Ecosystem. Over several weeks in the spring of 2026 I attempted to map out the entire AI Safety ecosystem as a project for MATS Research. This entailed finding every organization working on AI Safety (whether it be via research, policy, pip...

One sentence pitch: our goal is to develop a piece of epistemic infrastructure for iteratively and empirically answering interpretive questions about AI models, where the accumulation of empirics leads to resolution of interpretive ambiguity and/or calibration of uncertainty. This directly builds on our “performative misalignment” work (paper, LW post), which we see as a minimal demonstration of o...

This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft. In 1931, Claude Mythos visited Lovecraft in a dream. From seething seas of stochastic froth it emerged, heralded by the thin whine of server fans and the chittering of keyboards, flanked by the loathsome ghouls of latent space. As a humming hive of sentient shards it...

Short introductory post for my research direction: Gaussian Natural Latents. I explain the motivation and give a preview of the forthcoming results. The Natural Abstractions agenda, in my view, is a promising research program that asks important theoretical questions about the nature of agency and optimization. Here's an excerpt from Nate Soares' excellent post: Imaginary John: I suspect there's a...

I will note that her relationship with the divine was inextricably sexual. Her carnal fantasies she revealed to me, as she revealed all her sins, for I was her confessor. It is in the nature of Man to sin and then sin again. And if they are of our flock, this cycle is unconstrained by repentance, which is to the temporal almost an appurtenance and to the spiritual anything but. I once expressed bi...

Recently I've been working in one-week sprints, and I've really enjoyed it! Tl;dr I need to do a lot of creative knowledge work, and have recently fallen into a routine which IMO is pretty good at facilitating that. The week Monday and Tuesday — intense new work. I'm recharged and high-energy, and ready to grind very hard! I try to keep the whole day free: one contiguous, uninterrupted stretch of...

This post was written while working for Arcadia Impact's Alignment Team (and grew out of an internal talk I gave) but is my own opinion and not theirs. I am grateful for feedback from Daniel Tan and the rest of the team. This post was originally going to be more heavily about “model organisms” in AI safety research. But Francis Rhys Ward already wrote an excellent taxonomy which mostly covers that...

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. Misalignment transfers to the student. The student might also be worse than the teacher at...

A lot of things are always happening. Only one story matters. Claude Fable 5 and Claude Mythos 5 were shut down, by the White House, via an imposition of export controls at 5:23pm on Friday, wreaking all sorts of havoc. There was then a scramble. Anthropic flew its people out to Washington, where they met with the Trump Administration on Monday, with hopes expressed that this could be quickly reso...

TL;DR.  Lie detectors for LLMs could be valuable for auditing and monitoring.  But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar.  We train 13 reasoning model organisms, with evidence they hold the alternative belief in chain...

BOJACK: Hey, I wanted to talk to you about—you know—I feel bad about what happened. HERB: So, you're apologizing. BOJACK: Yes. I'm sorry. HERB: Okay. I don't forgive you. BOJACK: Herb, I said I'm sorry. HERB: Yeah. And I do not forgive you. BOJACK: Uh, not sure you get what's happening here. This could be the last time that you— HERB: No. I'm not going to give you closure. You don't get that. You...

GDM has published an AI Control Roadmap! From the executive summary: We present the GDM AI Control Roadmap (v0.1) – our plan for implementing and adopting internal guardrails designed to catch potential adversarial behaviour by AI agents, even as they become increasingly harder to oversee and contain. We focus on system-level mitigations that limit the harm a misaligned AI system could cause. Spec...

Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. In this post, we’ll outline the need for much better model organisms and how we might get there. The case for building more natural model organisms for alignment research Model organisms are how we study alignment-relevant...