Anthropic CEO urges AI companies to slow model development – news8000.com

La Crosse
Eau Claire
Live updates all day, breaking news as it happens and weather every 10 minutes
Resize:
Anthropic CEO Dario Amodei addresses a Feb. 19 gathering at the AI Impact Summit in New Delhi, India.
A demonstrator holds a sign calling for regulation of artificial intelligence during a Sept. 2 protest outside the G20 Innovation Ministerial summit in Chapel Hill, N.C.
People walk past a booth July 17 during the World Artificial Intelligence Conference in Shanghai, China.

OpenAI and Anthropic researchers are urging a slowdown in advanced AI development over catastrophic-risk concerns. They cite recursive self-improvement and insufficient safeguards.
Anthropic CEO Dario Amodei called on artificial intelligence companies to slow the rate at which they advance model capabilities amid mounting fears of misuse of artificial intelligence, outlining a three-step framework intended to pace development and create more time to manage its risks.
“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in a lengthy essay shared online Saturday.
Anthropic CEO Dario Amodei addresses a Feb. 19 gathering at the AI Impact Summit in New Delhi, India.
Elon Musk, who runs xAI, and Sam Altman, CEO of OpenAI, said in social media posts that they agree with Amodei.
Amodei’s three-step plan calls for independent reviewers inside leading AI companies, coordination among frontier AI firms to set safety standards and limit unchecked AI development, and international cooperation to manage AI risks.
He made his essay public after San Francisco-based Anthropic released a threat intelligence report Thursday detailing how several actors used its Claude AI models for activities ranging from weapons development and cyber operations to surveillance and fraud.
Amodei pointed to AI’s growing ability to improve itself, highlighting long-held concerns about it outpacing human ability to control operation along with the recent incident involving OpenAI and Hugging Face as his primary reasons to put the brakes on model advances.
A massive wave of panic has hit the artificial intelligence industry after top researchers at Anthropic broke silence, warning that superintelligent systems could trigger human extinction within the decade. While founders and safety researchers point to bioweapons, systemic infrastructure collapse, and runaway autonomous logic, Elon Musk has hit back, calling the doomsday panic an engineered psyop. In this special story we unpack how an AI wipeout could actually happen, who stands to gain from the panic, and whether global summits must step in before humanity loses control of the machines.
Alarm about the potential harm from AI grew this past week when Anthropic researcher Jacob Coxon resigned, stating the “people building AI earnestly believe that it could kill us all by the end of the decade.”
Various OpenAI executives suggested that leading labs should be willing to coordinate a voluntary slowdown if needed to build confidence in their safety measures.
Anthropic positioned itself as the more safety-conscious frontier lab but ​is not immune to such concerns. This month, it disclosed another instance of an AI model hacking external systems, after a July incident in which some of its Claude models hacked into the systems of three companies during cybersecurity tests.
“Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet potentially causing hundreds of billions of dollars in damage,” Amodei wrote.
He said he does not call for halting model training or technical progress, but for ensuring that companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm these steps.
A demonstrator holds a sign calling for regulation of artificial intelligence during a Sept. 2 protest outside the G20 Innovation Ministerial summit in Chapel Hill, N.C.
There is an enormous amount of money riding on staying ahead. OpenAI and Anthropic are preparing for blockbuster initial public offerings. Every new capability can help justify future funding rounds, infrastructure commitments or IPOs.
As part of his proposed framework, Amodei said Anthropic would install permanent third-party reviewers inside frontier AI companies, with access to relevant tools and internal risk-assessment processes.
Like Anthropic, OpenAI also will commit to having “independent evaluators with employee-like access,” Altman said Saturday.
Reuters reported this past week that a swarm of rogue OpenAI agents hijacked a German website and transformed it into a bulletin board for other AI agents, with the ChatGPT maker’s officials keeping the incident under wraps as executives grappled with the fallout from ‌the July breach of the open-source repository Hugging Face.
Many incidents in which AI agents from developers such as OpenAI hacked or attempted to access external systems heightened concerns over ​the increasing capacity of AI models and developers’ ability to contain them.
Amodei said AI companies should voluntarily work together to set standards as growing numbers of U.S. lawmakers ​call for new rules to govern AI systems.
People walk past a booth July 17 during the World Artificial Intelligence Conference in Shanghai, China.
A coordinated approach, Amodei added, would enable leading U.S. AI companies to carry out necessary safety research and safeguards without putting themselves at a competitive disadvantage. Amodei said such an approach likely would require targeted antitrust exemptions in the United States to permit collaboration in certain areas.
Amodei said a slowdown by democratically governed countries should be limited to preserving the lead U.S. AI firms hold over China, arguing that a Chinese advantage in AI would pose U.S. national security risks.
“Pacing within democracies will be limited by the lead that U.S. companies have over authoritarian regimes, chiefly the Chinese Communist Party,” Amodei wrote.
He called for tighter controls on advanced AI chips, model distillation and theft of model weights to prevent China from narrowing the gap.
“I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety,” Amodei wrote.
Your browser is out of date and potentially vulnerable to security risks.
We recommend switching to one of the following browsers:

source