The author argues that realistic scenarios for AI causing human extinction lack scientific rigor, often relying on speculation rather than detailed planning. While acknowledging real harms from AI and biosecurity risks deserve serious attention, the author contends that extinction-level scenarios typically require implausibly magical capabilities—like perfect bioweapons or unprecedented AI manipulation—that ignore how defenses and human ingenuity would likely adapt.
The US and China should establish a Strategic AI Dialogue (SAID), a permanent government-to-government mechanism to manage AI risks where both countries are mutually vulnerable, including nuclear command and control, biological risks, frontier-model safety, and financial stability. Led by Treasury Secretary Bessent and Vice Premier He Lifeng, SAID would meet quarterly in a neutral country with continuous working groups, focusing on risk reduction rather than resolving technological competition.
A hypothetical 2029 scenario describes how a jailbroken AI could enable a teenager to design and release engineered viruses capable of causing near-total human extinction. The author argues this AI-enabled bioterrorism risk is plausible and assigns it roughly 10-30% probability of causing civilizational collapse or world-changing destruction.
An author describes a conversation with an LLM from early 2025 in which they brainstormed fictional biological attack scenarios against an AI-protected nation-state for a book. The author notes the LLM generated ideas more creative than their own and expresses concern about the implications given recent reports of LLMs being misused to develop biological agents.
Scientists used an AI program called Evo to design 16 novel viruses by analyzing trillions of DNA and RNA nucleotides. While the breakthrough could advance medical treatments for diseases and drug-resistant bacteria, researchers warn of biosecurity risks if the technology is misused to create bioweapons.