As frontier AI accelerates, tech leaders are debating the likelihood of p(doom): ranging from a complete annihilation scenario, to human disempowerment via superintelligence. Here’s what you need to know.
In a recent interview on The Ezra Klein Show, Microsoft co-founder Bill Gates addressed his deep concerns about Artificial General Intelligence, calling it “the most dangerous thing that humans have ever gone near.”
Gates has been issuing warnings of late, about what he believes are near-term risks: AI overstepping biosecurity and cybersecurity red lines.
“We crossed the cyber and bio thresholds earlier this year,” he told Klein this month. “The ability for a cyberattack to scramble all the bank accounts, shut down the electric grid, that exists today. AI has crossed the threshold to empower a bioterrorist to kill hundreds of millions.”
Gates’ comments are of particular note because they frame AGI as a tool that can “empower” bad actors who mean to cause harm. It may not be the frontier AI companies that pull the trigger, Gates posits, but they have created the ecosystem and tools that can facilitate others to cause mass destruction.
The conversation about the probability of societal ‘doom’ has become commonplace in Silicon Valley and in innovation circles around the world. While Gates hasn’t weighed in with a ‘p(doom)’ value, numerous tech heavyweights have.
Deconstructing ‘doom’: Annihilation vs. disempowerment
P(doom) is a Bayesian probability estimate that the development of AGI, or superintelligence, within a 20-to-100-year window, will result in civilisational catastrophe. It is calculated on a percentile from 0 to 100.
AI safety researchers put p(doom) into two buckets: the first, and more extreme, is human annihilation. In the other doom bucket is that human “operational control” will permanently end during AGI, because software is acting at speeds beyond human cognitive capacity, and humans will be unable to redirect or decommission the governing AGI systems.
At one end of the spectrum is MIT Physicist and Future of Life Institute President Max Tegmark, who recently estimated p(doom) above 90 per cent. The ‘Godfather of AI’ Geoffrey Hinton suggested the value sat at less than 50 per cent, while LinkedIn founder Reid Hoffman puts it at 20 per cent.
Dario Amodei, the co-founder of leading AI lab Anthropic, said in 2023 that there is a 75 to 90 per cent chance that AI will NOT destroy humanity. Elon Musk has given estimates in the 10 to 20 per cent range too, noting that with “the advent of superintelligence, it is very difficult to predict what will happen next. There’s some chance that it will end humanity.”
OpenAI and Anthropic pretraining researcher Jacob Coxon stepped away from Claude’s parent company in September, noting that many AI insiders fear how quickly the technology could reshape and dismantle society. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote on X, echoing Gates and others concerns.
Anthropic Alignment Science lead Evan Hubinger says while the risk of superintellignece from current models is low, the rapid development of recursive self-improvement – where advanced AI teaches itself and consistently becomes exponentially smarter – is a very real risk.
“Jacob is correct here,” Hubinger, who still holds his job with Anthropic, wrote on X in the wake of Coxon’s resignation. “We really do earnestly believe AI could kill all humans ! I personally think it is >10% within the next decade.”
Tempering the doom
Jensen Huang, founder of NVIDIA which benefits greatly from the proliferation of AI, Marc Andreessen, a self proclaimed ‘techno-optimist’ billionaire who is heavily invested in AI, and Yann LeCun, formerly the Chief AI Scientist at Meta, all put p(doom) at around zero. The latter, called it BS in 2024.
Another researcher who resists reducing p(doom) to a number, is Australian Professor Toby Walsh, the Chief Scientist at UNSW AI Institute, and author of God AI: Boom or Doom? What to Expect When Machines Outsmart Us.
Walsh states in the book that “there are clear risks that powerful AI threatens our very existence as a species,” but in a recent interviews tempers the doomsday rhetoric by calling a global annihilation event from hypothetical biopathogens as “fanciful.”
Sam Altman warned about the inevitability of superintelligence more than a decade ago, cautioning that regulation was needed to prevent “malevolent humans intentionally misusing machine intelligence to, for example, wreak havoc on worldwide financial markets or air traffic control systems.” Of late, he refuses to be anchored to a specific p(doom) figure, but is on the record as saying it is not zero.
Former Google Deep Mind chief Demis Hassabis also questions the rationale – and science – behind p(doom) values and says there is not enough precision for him to make an estimate.
“I don’t have a p doom number. I think it’s a kind of a little bit of a ridiculous notion because what I would say is it’s definitely non-zero and it’s probably non-negligible. So that in itself is pretty sobering,” Hassabis told Lex Friedman in 2025.
Now Deep Mind’s Chairman, alongside his new role as Alphabet’s Chief Scientist, Hassabis is asking himself the questions so many regular people are.
“My view is it’s just hugely uncertain, right? What these technologies are gonna be able to do? How fast are they gonna take off? How controllable are they going to be?”
Forbes Women is a global network of female business owners, entrepreneurs and changemakers. Sign up to Forbes Women here
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
