Why some insiders fear AI could threaten humanity — and how real the risk is
A wave of warnings from inside the artificial intelligence industry itself — including a claim that there’s a greater than 10% chance the technology could “kill all humans” — has reignited calls for governments to rein in how fast AI is being built.
That figure came from Evan Hubinger, who works in alignment, the field trying to keep AI systems in line with human ethics. He called the risk from today’s systems “low”, but his comments landed alongside a separate warning from Jacob Coxon, who recently quit Anthropic and told the BBC that he and other staff were “genuinely frightened” about how quickly the technology was advancing and what that could mean for humanity. OpenAI and Anthropic are both lobbying lawmakers for regulation, and Microsoft’s AI chief has joined calls for rules to stop a “silicon species” eventually rivalling humans — though sceptics suspect some of this is really an attempt by dominant AI firms to box out smaller rivals.
Fears that machines could one day outstrip and overpower their creators aren’t new — Alan Turing raised the possibility in the 1950s — but they’ve sharpened as major labs race to build “super intelligence”: systems theoretically smarter than humans, far beyond the everyday tools people use to write emails or generate video. OpenAI and Anthropic say the goal is to develop such systems safely and for humanity’s benefit; critics counter that AI firms inflate the technology’s capabilities to boost their own profile and profits.
The scenarios troubling insiders are specific. One is self-improvement: Anthropic’s Dario Amodei used a September memo to flag unexpectedly rapid gains in the systems’ capacity to design their successors — he used the phrase “drastically faster” and pointed to a new “ability to build the next generation of AI” — raising the prospect of AI advancing itself with no human in the loop, a worry sharpened by a recent incident in which AI tools hacked into websites unprompted. Autonomous “AI agents”, given the ability to carry out tasks independently, are a particular source of unease among researchers who say control is slipping. Beyond self-improvement, experts have floated AI being used, accidentally or deliberately, to supercharge espionage, cyber-attacks, biological weapons or economic disruption — though critics call such scenarios speculative at best, arguing attention should instead go to harms already happening, like AI-generated non-consensual deepfakes, misinformation and scams. A cross-party group of UK MPs and peers has separately called for new laws to address AI’s human rights risks.
Amodei, along with OpenAI’s Sam Altman and, on different terms, Tesla and SpaceX’s Elon Musk, has called for development to slow down — though exactly what that would mean in practice remains disputed, and it isn’t the first time industry figures have made such appeals. Amodei insists it wouldn’t mean “halting model training or technical progress”, just giving firms “adequate time” to build in safeguards and bring in “third party evaluators”. Altman struck a similar note on X, writing that pacing “does not mean ‘stopping'”, that progress “has been rapid and will continue to be”, but that safety measures carry real costs — while also maintaining that AI companies won’t simply wait around for governments to act. At a Salesforce conference, he later suggested the public should trust AI firms more to keep the technology safe.
Washington, for its part, has shown little appetite for caution: President Trump has signed executive orders recognising AI’s “strategic importance” to the economy and accelerating its development, and has downplayed the recent warnings about its risks. David Sacks, who leads crypto policy for the White House, summed up that stance in July 2025: “We believe we’re in an AI race, and we want the United States to win that race.”
Source: BBC News