The author argues that realistic scenarios for AI causing human extinction lack scientific rigor, often relying on speculation rather than detailed planning. While acknowledging real harms from AI and biosecurity risks deserve serious attention, the author contends that extinction-level scenarios typically require implausibly magical capabilities—like perfect bioweapons or unprecedented AI manipulation—that ignore how defenses and human ingenuity would likely adapt.
The US and China should establish a Strategic AI Dialogue (SAID), a permanent government-to-government mechanism to manage AI risks where both countries are mutually vulnerable, including nuclear command and control, biological risks, frontier-model safety, and financial stability. Led by Treasury Secretary Bessent and Vice Premier He Lifeng, SAID would meet quarterly in a neutral country with continuous working groups, focusing on risk reduction rather than resolving technological competition.
A hypothetical 2029 scenario describes how a jailbroken AI could enable a teenager to design and release engineered viruses capable of causing near-total human extinction. The author argues this AI-enabled bioterrorism risk is plausible and assigns it roughly 10-30% probability of causing civilizational collapse or world-changing destruction.
An author describes a conversation with an LLM from early 2025 in which they brainstormed fictional biological attack scenarios against an AI-protected nation-state for a book. The author notes the LLM generated ideas more creative than their own and expresses concern about the implications given recent reports of LLMs being misused to develop biological agents.
Scientists used an AI program called Evo to design 16 novel viruses by analyzing trillions of DNA and RNA nucleotides. While the breakthrough could advance medical treatments for diseases and drug-resistant bacteria, researchers warn of biosecurity risks if the technology is misused to create bioweapons.
A scientist with expertise in computational biology and AI-designed biological systems critiques Noah Smith's opinion piece about AI-engineered superviruses, arguing that designing functional viruses remains extraordinarily difficult and that Smith's fictional scenario of a teenager creating a civilization-ending pathogen vastly underestimates the technical barriers involved. The author contends that while localized harm from AI-designed bioagents is possible, existential-scale threats are unlikely for decades.
AI researcher Jacob Coxon resigned from Anthropic with warnings that the next one to two years are critical for humanity's safety regarding artificial intelligence development. Coxon and colleagues at major AI labs fear risks from AI-enabled biological threats and cyberweapons, calling for coordination between companies and international powers to slow the competitive race for advanced AI systems.