Anthropic has called for a coordinated global pause in artificial intelligence development, warning that humans risk losing control of the technology. The company is described in the original report as the world’s most valuable AI start-up.
The company behind Claude offered to suspend work on more powerful systems if it could be confident that others would do the same.
Company executives wrote that, if the technology’s development could be slowed effectively to allow more time to address its enormous implications, they believed that would probably be beneficial.
They compared the rise of powerful AI systems to an arms-control problem and warned that there was limited time to bring the technology under control.
The original report states that Anthropic recently overtook ChatGPT developer OpenAI in valuation, reaching $965 billion, or £719 billion. It says the company has withheld its most powerful system, Mythos, from the general public over concerns that it could enable devastating cyberattacks.
In an essay published on Thursday evening, 4 June, Anthropic research leader Marina Favaro and company president Jack Clark said AI was approaching the point at which systems would develop themselves, according to reporting by The Telegraph.
They said this stage, known as recursive self-improvement, could trigger an explosion in capabilities but might also leave humans unable to control AI systems.
Anthropic has positioned itself as particularly committed to AI ethics and safety, while pursuing research breakthroughs that it acknowledges could lead to dangerous systems.
Chief executive Dario Amodei has said there is a 25% chance that things could go very badly.
On Thursday, the company cautioned that a slowdown which merely allowed less careful actors to catch up could leave everyone less safe.
It argued that the world should have the option to slow or pause frontier AI development so that social institutions and alignment research could keep pace.
However, it warned that coordinating a worldwide pause would be exceptionally difficult, pointing to intense AI competition between the United States and China.
A meaningful slowdown or pause would require multiple well-resourced laboratories at or near the technological frontier, across several countries, to agree to stop under the same conditions, the company said.
It would also require each participant to verify that the others had actually stopped. AI training runs, it noted, are much easier to conceal than missile silos.
Favaro and Clark said Anthropic staff were now producing eight times as much code as during 2021–2025. AI was also improving at suggesting ideas and designing research, they said, bringing systems that build themselves a step closer.
Other experts have accused the company of exaggerating AI’s capabilities or issuing warnings to encourage regulations that could disadvantage its competitors.
The warnings followed a sharp fall in AI chip shares after semiconductor company Broadcom disappointed investors with its sales forecasts.
Broadcom shares fell 13%, wiping around $300 billion from its market value, while shares in fellow chip companies Micron, SK Hynix and Arm also plunged.
The sell-off fuelled fears of an AI bubble as investors questioned demand for Broadcom’s chips.
