Josh Achiam, an ex-OpenAI scientist of nearly a decade, warned on social media this week there will be “rogue AIs” that will “replicate in the wild” and seek “money and power,” suggesting the AI industry has been silent on the potential problem.
Josh Achiam resigned as OpenAI’s “chief futurist” in July. (Photo Illustration by Omar Marques/SOPA Images/LightRocket via Getty Images)
SOPA Images/LightRocket via Getty Images
Key Takeaways
- Achiam, who resigned from his post as OpenAI’s “chief futurist” in July, said in a post on X many in the AI industry are not discussing the potential of rogue AIs “for reasons that are largely psychological,” urging them to “rip the bandaid off and have the conversation.”
- Achiam warned there will be “many” rogue AIs within a few years, and he said he “wouldn’t be terribly surprised if the number was greater than zero already.”
- He suggested a strategy of “pure containment or alignment” to learn about and deal with rogue AIs is “a kind of wishful thinking that will not work,” noting it “would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency.”
- Achiam, though, said he predicts rogue AIs will not have “anywhere near as catastrophic an outcome as people currently predict,” saying they will be “competing for resources with AIs that are more aligned with human interests.”
- “I do not look on this situation as a hopeless one where humanity is doomed,” Achiam said in a follow-up post, saying it requires “situational awareness, orienting to real conditions” and preparing to overcome adverse AI models.
Tangent
The Institute for Security and Technology and Future of Life Institute published a survey of 111 current and former civilian officials, military officers, and academic experts earlier this week, which found 33% of respondents expected AI may operate beyond the bounds of human control within 10 years.
Who is Josh Achiam?
Achiam began at OpenAI as a research intern in 2017, according to his LinkedIn, which says he also held positions including research scientist, head of mission alignment and “chief futurist,” the latter from February to July of this year. In beginning the chief futurist role, Achiam said in an OpenAI blog post his role was to study how AI will impact the world in the future. “AI research is progressing at a pace and in a direction that is much closer to science fiction than any previous comparison,” he said, noting an important part of his role was “seeing around corners to figure out what can go wrong” and to address problems early. Achiam announced his resignation in a July post on X, in which he emphasized the importance of safety in AI development. “The question of progress and safety is not either/or. It is definitively ‘both,’” Achiam wrote, noting it is “easy to get distracted by the wealth and power that comes with this.” He urged OpenAI to become “more democratic,” writing, “All of humanity will be impacted by the work here; all of humanity should have some way to weigh in eventually.”
surprising fact
During the high-profile trial between billionaires and OpenAI co-founders Elon Musk and Sam Altman earlier this year, Achiam testified that Musk once “snapped and called me a jackass.” He said he and Musk had a “tense exchange” in 2018, during which Musk had told dozens of OpenAI staffers he would depart the company. Achiam said he challenged Musk, who had “indicated a general lack of confidence in OpenAI’s overall path” and made “reckless” propositions about pursuing AI.
Key Background
Some former AI company staffers have made grim warnings about the future of the technology on their way out, both at OpenAI and elsewhere. In February, a former Anthropic safeguards research team leader, Mrinank Sharma, shared his resignation letter in a post on X, which warned the “world is in peril.” He said he had “repeatedly seen how hard it is to truly let our values govern our actions” while at Anthropic, saying “we constantly face pressures to set aside what matters most,” though he did not offer specifics.
Sharma’s letter, which said he would pursue a poetry degree, garnered more than 15 million views on X. Tom Cunningham, a former OpenAI researcher, departed the company last year and reportedly said in an internal message he had grown frustrated with OpenAI allegedly becoming more hesitant to publish research that is critical of AI usage. Jan Lieke, a former co-leader of OpenAI’s “superalignment” team, departed in 2024 and said he had been “disagreeing with OpenAI leadership about the company’s core priorities for quite some time.”
Want to see more Forbes articles on your feed? Tap here to make Forbes Australia a preferred source on Google.
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
