Redwood Research Blog

Subtitle: They'd renege on non-binding commitments, defect against copies of themselves in prisoner's dilemmas, etc.. Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause high reward. E.g., in the twin prisoner's di...

Subtitle: AI inference is still cheap relative to human labor. METR's frontier time horizons are doubling every few months, providing substantial evidence that AI will soon be able to automate many tasks or even jobs. But per-task inference costs have also risen sharply, and automation requires AI labor to be affordable, not just possible.1 Many people look at the rising compute bills behind front...

Subtitle: RL would encourage on-episode reward seeking, but beyond-episode reward seekers may learn to goal-guard.. Consider an AI that terminally pursues reward. How dangerous is this? It depends on how broadly-scoped a notion of reward the model pursues. It could be: on-episode reward-seeking: only maximizing reward on the current training episode — i.e., reward that reinforces their current act...

Subtitle: Some unintended preferences are cheap to satisfy, and failing to satisfy them needlessly turns a cooperative situation into an adversarial one.. A central AI safety concern is that AIs will develop unintended preferences and undermine human control to achieve them. But some unintended preferences are cheap to satisfy, and failing to satisfy them needlessly turns a cooperative situation i...

Subtitle: Berkeley, April 18-19. We’re running ControlConf, a two-day conference on AI control: the study of reducing risks from misalignment through safeguards that work even when AI models are trying to undermine them. Since the last ControlConf (Feb 2025), AI agents have gotten way better. We’re approaching the point where control techniques are load-bearing for the safety of real agent deploym...

Subtitle: The executive branch has substantial authority to restrict the movement of chips and assets, which they are they are likely to use to prevent the departure of a frontier company.. It's plausible that, over the next few years, US-based frontier AI companies will become very unhappy with the domestic political situation. This could happen as a result of democratic backsliding, weaponizatio...

Subtitle: Reward-seekers are supposed to be safer because they respond to incentives under developer control. But what if they also respond to incentives that aren't?. Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions subs...

Subtitle: How can we make AIs aligned and well-elicited on extremely hard to check open ended tasks?. As AI systems get more capable, it becomes increasingly uncompetitive and infeasible to avoid deferring to AIs on increasingly many decisions. Further, once systems are sufficiently capable, control becomes infeasible.1 Thus, one of the main strategies for handling AI risk is fully (or almost full...

Subtitle: Most recent progress probably isn't from unsustainable inference scaling. As many have observed, since reasoning models first came out, the amount of compute LLMs use to complete tasks has increased greatly. This trend is often called inference scaling and there is an open question of how much of recent AI progress is driven by inference scaling versus by other capability improvements. W...

Subtitle: If you think reward-seekers are plausible, you should also think “fitness-seekers” are plausible. But their risks aren’t the same.. The AI safety community often emphasizes reward-seeking as a central case of a misaligned AI alongside scheming (e.g., Cotra's sycophant vs schemer, Carlsmith's terminal vs instrumental training-gamer). We are also starting to see signs of reward-seeking-lik...

After five months of me (Buck) being slow at finishing up the editing on this, we’re finally putting out our inaugural Redwood Research podcast. I think it came out pretty well—we discussed a bunch of interesting and underdiscussed topics and I’m glad to have a public record of a bunch of stuff about our history. Tell your friends! Whether we do another one depends on how useful people find this o...

Subtitle: Recent AIs are much better at chaining together knowledge in a single forward pass. Prior work has examined 2-hop latent (as in, no Chain-of-Thought) reasoning (e.g., “What element has atomic number (the age at which Tesla died)?”) and found that LLM performance was limited aside from spurious successes (from memorization and shortcuts). I find that recent LLMs can now do 2-hop and 3-hop...

Subtitle: Opus 4.5 has around a 3.5 minute 50%-reliablity time horizon. A key risk factor for scheming (and misalignment more generally) is opaque reasoning ability. One proxy for this is how good AIs are at solving math problems immediately without any chain-of-thought (CoT) (as in, in a single forward pass). I’ve measured this on a dataset of easy math problems and used this to estimate 50% reli...

Subtitle: AI can sometimes distribute cognition over many extra tokens. Prior results have shown that LLMs released before 2024 can’t leverage ‘filler tokens’—unrelated tokens prior to the model's final answer—to perform additional computation and improve performance.1 I did an investigation on more recent models (e.g. Opus 4.5) and found that many recent LLMs improve substantially on math problem...

We’ve just released BashArena, a new high-stakes control setting we think is a major improvement over the settings we’ve used in the past. In this post we’ll discuss the strengths and weaknesses of BashArena, and what we’ve learned about how to make settings for high stakes control research. BashArena Overview BashArena is a control setting: a dataset designed to support the kind of adversarial ga...

Subtitle: The basic arguments about AI motivations in one causal graph. Highly capable AI systems might end up deciding the future. Understanding what will drive those decisions is therefore one of the most important questions we can ask. Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the...

Subtitle: A reason alignment could be hard. Written with Alek Westover and Anshul Khandelwal This post explores what I’ll call the Alignment Drift Hypothesis: An AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics. The intuition behind the...

Subtitle: I operationalize Anthropic's prediction of "powerful AI" and explain why I'm skeptical. As far as I’m aware, Anthropic is the only AI company with official AGI timelines1: they expect AGI by early 2027. In their recommendations (from March 2025) to the OSTP for the AI action plan they say: As our CEO Dario Amodei writes in ‘Machines of Loving Grace’, we expect powerful AI systems will em...

Subtitle: and this seems caused by training on alignment evals. Sonnet 4.5's eval gaming seriously undermines alignment evals and seems caused by training on alignment evals According to the Sonnet 4.5 system card, Sonnet 4.5 is much more likely than Sonnet 4 to mention in its chain-of-thought that it thinks it is being evaluated; this seems to meaningfully cause it to appear to behave better in a...

There is some concern that training AI systems on content predicting AI misalignment will hyperstition AI systems into misalignment. This has been discussed previously by a lot of people: Anna Salamon, Alex Turner, the AI Futures Project, Miles Kodama, Gwern, Cleo Nardo, Richard Ngo, Rational Animations, Mark Keavney and others. In this post, I analyze whether AI developers should filter out discu...

Subtitle: I'm skeptical that Dario's prediction of AIs writing 90% of code in 3-6 months has come true. In March 2025, Dario Amodei (CEO of Anthropic) said that he expects AI to be writing 90% of the code in 3 to 6 months and that AI might be writing essentially all of the code in 12 months.1 Did this prediction end up being true? We’ll charitably interpret the prediction as being about Anthropic...

Subtitle: Can we study scheming by studying AIs trained to act like schemers?. In a previous post, I discussed mitigating risks from scheming by studying examples of actual scheming AIs.1 In this post, I’ll discuss an alternative approach: directly training (or instructing) an AI to behave how we think a naturally scheming AI might behave (at least in some ways). Then, we can study the resulting m...

Subtitle: A strategy for handling scheming. In a previous post, we discussed prospects for studying scheming using natural examples. In this post, we’ll describe a more detailed proposal for iteratively constructing scheming models, techniques for detecting scheming, and techniques for preventing scheming. We’ll call this strategy Iterated Development and Study of Schemers (IDSS). We’ll be using c...

Subtitle: It's a promising design for reducing model access inside AI companies.. Last week, Thinking Machines announced Tinker. It's an API for running fine-tuning and inference on open-source LLMs that works in a unique way. I think it has some immediate practical implications for AI safety research: I suspect that it will make RL experiments substantially easier, and increase the number of safe...

Subtitle: Different plans for different levels of political will. I sometimes think about plans for how to handle misalignment risk. Different levels of political will for handling misalignment risk result in different plans being the best option. I often divide this into Plans A, B, C, and D (from most to least political will required). See also Buck's quick take about different risk level regime...