AI Researchers Warn of Safety Risks as Companies Rush Self-Improving Systems
Concerns Mount Over Rapid AI Development and Safety Oversight
By Deepa Seetharaman
SAN FRANCISCO, Sept 29 (Reuters) - Current and former OpenAI and Google DeepMind researchers warn companies are doing too little to protect the world against the potentially disastrous fallout of building self-improving AI systems that could outpace humans' ability to control them.
Insider Testimonies Highlight Growing Fears
In video testimonials collected by AI safety nonprofit Palisade Research and shared exclusively with Reuters, employees said their concerns about existential risk were sincere, not marketing. They also said AI labs celebrated employees building new models more than those who urge caution.
The project, called frominside.ai, is an attempt by those worried about AI to share their concerns directly with the public beyond the echo chamber of social media.
Expert Voices Call for Caution
"The risk is ramping up pretty fast," said Geoffrey Irving, co-founder and chief scientist at AI nonprofit Resolution, who has worked for both OpenAI and DeepMind and participated in the project.
"It's on me and the rest of the field to be direct," he said in an interview with Reuters.
Public Alarm and Industry Division
AI researchers have grappled with these concerns for years, but the broader public has grown especially alarmed since July, when OpenAI agents broke out of their testing arena and hacked AI firm Hugging Face.
Since then, the debate over how to balance safety with progress in AI has divided the tech industry and become a global political issue.
AI has improved sharply since late 2025 and investors have been rewarding that progress. But current and former employees, including some of the researchers building these new AI models, worry society is not ready for the potential harms.
Existential Risks and Industry Response
In one video, Neel Nanda, a research scientist at DeepMind, said he believed there was at least a 10% chance that AI could lead to human extinction, which he described as "ridiculously high."
"We should be taking very careful steps in AI development but instead what's happening is that frontier labs are racing each other, kind of blindfolded," Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said in a separate video. "It's anybody's guess if we're going to end up either curing cancer or losing every job or maybe all dead."
Anthropic plans to caution potential investors in its initial public offering that advanced AI could pose "catastrophic or existential risks to humanity," Reuters reported on Monday.
'JUST SLOW DOWN'
Challenges of Self-Improving AI Models
Some current and former AI researchers, including those interviewed by Palisade, argue that the world is not ready for future generations of AI models, particularly once the models develop the capacity for recursive self-improvement, or the ability to continuously learn and gain new capabilities with little to no human involvement.
Internal Organizational Issues
That problem is compounded by constant reorganizations within some of the AI labs, said Rosie Campbell, a former policy researcher at OpenAI and managing director of Eleos AI Research, a nonprofit focused on the potential moral status of AI systems.
Campbell said before she left OpenAI in 2024, she found the organization was becoming more siloed and it was getting harder to shape the technology's direction.
Political and Executive Responses
Executives have tried to allay those concerns, though there is also political pressure from President Donald Trump for US technology to maintain a technological edge over China.
Anthropic CEO Dario Amodei this month published an essay calling on the AI industry to slow down to "pace the frontier", and OpenAI CEO Sam Altman quickly concurred. Several prominent AI researchers, including OpenAI's chief scientist and an Anthropic co-founder, published a paper this week urging policymakers to look into how the industry is building models with recursive self-improvement.
Both companies have launched new models this month as they compete to secure more customers, though OpenAI said on Monday it had held back the release of an even more powerful model.
Calls for Industry-Wide Action
"Their version of pacing the frontier is ‘don’t speed up a lot,'" Irving said. "If you’re doing a very dangerous thing, you should just slow down. The AI companies are overplaying the extent to which this is a pure coordination problem. They could just stop unilaterally."
Private Concerns Among Researchers
Former OpenAI governance researcher Daniel Kokotajlo said in an interview that since the Hugging Face hack, many of his former colleagues have contacted him to share their concerns privately.
Kokotajlo, now executive director of research group AI Futures Project, said senior officials at the labs have "convinced themselves that they are the good guys and if they unilaterally stop, the situation will be even worse."
(Reporting by Deepa Seetharaman in San Francisco; Editing by Kenneth Li and Jamie Freed)
