Anthropic warns AI may soon begin recursive self-improvement

Leading companies in artificial intelligence (AI) might need to consider slowing their progress, according to one of the industry’s most dynamic players.

Anthropic, the company behind the Claude chatbot, has suggested that AI systems could soon reach a stage they describe as recursive self-improvement. This would enable AI systems to design and create their own successors with minimal human intervention, posing a potential risk of humans losing control over the technology.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Anthropic mentioned in a June 4 blog post titled “When AI Builds Itself.”

On supporting science journalism

If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

This proposal underscores a significant challenge in AI governance. Implementing a slowdown would require consensus among competing companies and governments worldwide, without any binding treaty, while competition is growing more intense. This makes Anthropic’s call both technically crucial and politically sensitive, as the company remains a leader in the AI race.

The rapid pace of technological development could have “huge implications” for society, according to the blog post. Anthropic cited its own operations as a warning, noting that Claude now generates more than 80 percent of the code integrated into its systems, a substantial increase from the low single digits before the launch of Claude Code in early 2025. The company also reported that its engineers now produce about eight times more code per quarter than a few years ago, indicating a diminishing human role in AI development. “We are not there yet, and recursive self-improvement is not inevitable,” Anthropic stated. “But it could come sooner than most institutions are prepared for.”

Anthropic proposed a “global coordination mechanism” to decelerate or pause AI development, allowing society to catch up.

The company offered few details and referenced arms-control agreements on intermediate-range nuclear missiles as a loose framework. For any slowdown to be effective, leading AI labs would need to participate, and there must be a reliable way to verify compliance.

“I don’t think it’s a genuine call to slow down,” said Noah Giansiracusa, an associate professor of mathematics at Bentley University and author of two books on algorithms and society. “We’ve read [Anthropic CEO] Dario Amodei’s blog posts. I think he wants to keep going full speed ahead.”

Anthropic did not respond to Scientific American’s questions about the practical implementation of such a slowdown or the criticism that it has exaggerated its systems’ capabilities.

Giansiracusa also views a pause as impractical. “It’s literally impossible,” he stated. “Zero chance there will be a slowdown. I’m not even talking China—Elon Musk would never slow down.”

The proposal follows a pattern that raises suspicion among some researchers. Two months ago, Anthropic unveiled a model called Mythos, which it chose not to release publicly, citing its effectiveness in identifying software vulnerabilities. The call for a slowdown also coincided with Anthropic’s confidential filing for an initial public offering and a recent funding round that valued the company near $1 trillion.

To skeptics, such dramatic announcements appear as a business strategy to attract regulatory attention to the cutting edge while Anthropic continues its rapid advancement. Mark Riedl, a professor in the School of Interactive Computing at the Georgia Institute of Technology, posted on Bluesky that “the big AI companies are all jumping on the ‘recursive self-improvement’ hype train.”

Anthropic plans to spend the next few months engaging with governments, researchers, and rival AI firms to explore whether a coordinated slowdown could be effectively implemented.

“I don’t really see the cause for concern,” Giansiracusa commented. “They’re flirting with the idea of the singularity—that it’s a game changer, and I just don’t see that. I see it continuing to progress. Maybe things will speed up; maybe it won’t.” According to him, the evidence Anthropic provides—more code written by AI—suggests the technology is useful, rather than representing “a great leap.”

It’s Time to Stand Up for Science

If you enjoyed this article, I’d like to ask for your support. Scientific American has served as an advocate for science and industry for 180 years, and right now may be the most critical moment in that two-century history.

I’ve been a Scientific American subscriber since I was 12 years old, and it helped shape the way I look at the world. SciAm always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does that for you, too.

If you subscribe to Scientific American, you help ensure that our coverage is centered on meaningful research and discovery; that we have the resources to report on the decisions that threaten labs across the U.S.; and that we support both budding and working scientists at a time when the value of science itself too often goes unrecognized.

In return, you get essential news, captivating podcasts, brilliant infographics, can’t-miss newsletters, must-watch videos, challenging games, and the science world’s best writing and reporting. You can even gift someone a subscription.

There has never been a more important time for us to stand up and show why science matters. I hope you’ll support us in that mission.