The debate over whether AI will trigger the end of the world moved from think tanks to living rooms last week. Among the warnings, hot takes, and spins were a confusing variety of opinions on what, exactly, the real problem is. Should we be worried about the stock market? A rogue agent releasing a virus? China making killer AI first? Or the environmental impact?
Unlike many hot-button social issues, the division of opinions on AI safety doesn’t map neatly along political lines or tax brackets. Instead, there are camps distinguished by worldviews and moral codes that have long existed in the San Francisco tech scene.
Among the groups — which include effective altruists and accelerationists — are well-funded movements with growing political sway and cultural capital. And they all have something to say on AI safety.
The debate has put some typically opposing forces on the same side. U.S. President Donald Trump and Chinese President Xi Jinping appear to agree on the path forward, along with Nvidia CEO Jensen Huang. But many of Huang’s CEO counterparts — including OpenAI’s Sam Altman and Anthropic’s Dario Amodei are pushing to slow development for safety reasons.
To help with the confusion, The Standard spoke with top AI thinkers and asked them to share their views, in their own words, on the doom fears and what happens next.
The Effective Altruist: AI doom is justified
Malo Bourgon, CEO of the Machine Intelligence Research Institute
The current AI-doom fears are unfortunately spot-on — the world is finally waking up to a danger that’s been creeping closer for decades. The companies are bringing it up now mostly because they can’t ignore it or sweep it under the rug any longer. This past summer saw multiple public disasters in which swarms of AIs hacked their way out of various companies and wreaked havoc on the broader internet.
Our biggest fear is that, in the very near future, one of these companies will set up an autonomous agent that is capable of developing a smarter agent that is capable of developing an even smarter agent, and the whole thing will spiral out of control. Once significantly smarter-than-human AI exists, the game is essentially over — there’s no putting the genie back in the bottle. The people building it know exactly how dangerous it is, but they’re stuck in a race that they can’t escape, because if any one group stops, the next less-responsible group will race ahead.
The Short-Term Alarmist: AI doom is a distraction
Dylan Baker, lead research engineer at the Distributed AI Research Institute
These kinds of conversations about an AI “kill switch” or existential risk position all of the harms of AI in some speculative future, and it distracts from the harms of AI that are currently happening. It also positions tech companies and tech CEOs as the only people capable of regulating or stopping these harms. The way it is framed is about marketing and company speculation ahead of IPOs and less about concrete harms, of which there are many. There’s the environmental impact of data centers, the labor harms of people training these models, teen and child safety, mental health, stealing artists’ work, the list goes on.
The real way for us to ensure we have a safe future is to stop treating AI as exceptional and impose regulations and safeguards, like any other industry. Much like food safety, where you have everything from the FDA to restaurant safety standards, I think regulation is going to look like a lot of different things working in concert. To start, we need to regulate surveillance infrastructure and data center construction. Those things are harder to get rid of once they are in place.
The Politician: Get ahead of doom with regulation
State Sen. Scott Wiener, candidate for Congress
Government must act. We can’t rely on industry self-regulation. The solution must include both industry changes and government regulation. California should lead in responding to this moment by strengthening the AI guardrails we pioneered for the country. We have the expertise and the political will to take action here, and we can set strong safety standards while Washington gets it together. (We can even require kill switches.) The president is spouting nonsense that rogue AI is a hoax right now, but he has been known to change his tune, and we should keep pushing to ensure that he does.
I’ve been advocating to get ahead of AI risks since 2023, when I introduced the first AI safety legislation in the country. This technology has the potential to provide tremendous benefits to the world, but we also have a responsibility to get ahead of severe risks we’ve been repeatedly warned about. I pursued that legislation — focused on catastrophic risk — in 2024 and 2025. The first version, which included a kill switch and liability for reckless release of dangerous models, was vetoed. The second version, focused on transparency by the big AI labs and reporting of critical incidents, was signed into law. But we need to do more.
The Accelerationist: The best defense is going full speed
Josh Constine, venture capitalist at SignalFire
I’m a Hobbesian accelerationist. We need to go full speed on development while being cautious with deployment, because humans are fallible in their application of AI.
There’s no way to slow down China or open source, so having the strongest AI is likely the best defense against other nations’ AI, quantum decryption, and cyberattacks. What we should consider slowing down is the public release of models that lack proper evaluation and safeguards in order to prevent misuse of AI for political propaganda, biological and chemical weapons research, cybersecurity attacks, and more. Some people are simply deranged enough to want to inflict suffering on us all. Combined with the democratization of violence over time allowing each individual to have more destructive power as we’ve evolved from fists and rocks to guns and tanks to nukes and germs, unchecked AI deployment could enable supervillainy.
Right now, we absolutely need stronger prerelease safety monitoring, and that may become problematically expensive as the tasks evaluated become more complex. We also may need a narrow ban on the creation of models and reinforcement learning training environments dedicated to cataclysmic purposes like designer viruses. But a broad slowdown of AI development risks hobbling the defenders of democracy and civilization.
The Existential Risk Skeptic: Accidents happen; extinction won’t
Shreeharsh Kelkar, interpretive social scientist at UC Berkeley
The current AI doom fears are exaggerated. If the idea is that AI will kill all humanity, sorry, but it’s not going to happen. Humanity is pretty tough. We’ve survived worse, including two world wars, plagues, and much else.
If the idea is that some accident will happen that will kill a few (or many) people, that is not an unreasonable worry. Accidents do happen. We live, after all, in a mass society that is full of complex systems that interlock together and whose whole is more than the sum of their parts. Planes crash, factories pollute; these are all complex systems that need to be fine-tuned, and it’s something we need to be doing all the time. It’s not work that ever ends. It is a mistake to think of this as some kind of special “misalignment” problem that is unique to AI; indeed, all complex systems have alignment issues, and those have to be worked on continuously. It is these accidents we should fear.
The Analyst: Regulation won’t slow AI
Aniket Shah, head of the Washington policy and sustainability research teams at investment bank Jefferies
There are a whole bunch of conspiracy theories around why this conversation is happening right now. I don’t conjecture on that, but when a thousand researchers at frontier model companies sign a letter saying, “We’re concerned,” you should probably listen to that. That’s not a thousand people all in it together to make sure their IPO is successful. It’s actually people who are close to the technology worried about what they are seeing.
There are risks, but AI is also underpinning U.S. economic growth: 60% to 70% of GDP growth is tied to the AI investment boom. The only way these concerns around risk and safety actually translate into real intervention that could slow down AI development is if there is some catastrophic-type event — what Three Mile Island, Chernobyl, or Fukushima was for nuclear. The Hugging Face incident isn’t enough. It would have to be much more significant for politicians to intervene in a way that would slow down GDP growth.
All the things that are being discussed from a regulatory perspective are actually quite limited. There’s a narrative out there that if Democrats win in the midterms, you’re going to see a lot of AI regulation, and that’s going to slow down AI. I don’t see that happening, because even the most provocative state-level AI regulations are actually quite benign. They’re good to do, but they’re not going to slow the pace of AI development at all.
The Protestor: Rise up
Michaël Trazzi, founder of the Stop the AI Race movement
Since the tweets by Jacob Coxon last week announcing his departure from Anthropic, the world is waking up. It’s been the first week in human history where humanity started noticing and deeply reflecting on the risks from AI, and also when the public learned for the first time that these companies are risking all of our lives to push for a technology that they cannot control. We organized a protest Thursday morning, where we walked from OpenAI to Anthropic to City Hall. We invited the employees to quit their jobs and join us.
The Researcher: The inflection point is now
Christopher Ackerman, former Google researcher turned AI safety scholar
It’s kind of amazing to see AI safety breaking into the mainstream after all this time of it being ignored or considered crazy by a lot of society. The companies have talked publicly about it for years, but objectively, the models haven’t been worrisome enough for that to break through. The Hugging Face incident was this inflection point that people could rally around and say, “Yeah, we need to do something.”
The biggest risk to this is that it becomes partisan, because then it’s just this constant battle. It’s great news that some Republican congressmen and people running for office have spoken out, in addition to the large number of Democrats. Unfortunately, the Trump administration and various online allies are taking a fairly aggressive approach and being dismissive of these concerns. My biggest fear is that it turns into the COVID vaccines in reverse, where Republicans just decided back then it was really dangerous. I hope that Trump and President Xi will talk about it. Perhaps in the new Congress we might see some legislation.
The Humanist: Don’t give up agency
Imran Khan, project director for psychosocial evaluations of AI at the Center for Humane Technology
My biggest fear is that we’re being incentivized to hand over our agency to AI. I worry we’ll stop thinking and making decisions for ourselves, and stop imagining and articulating what society should look like — because the system pushes us to “just ask AI” instead. And I worry that this process has already started. But we can stop it — if we collectively decide that we prefer AI that is optimized to maximize our welfare, well-being, and freedom, rather than simply our engagement. After all, this will put us in a better position to respond well to existential risks — from AI and anything else.
The Anthropologist: AI doom is telling us something bigger
Caroline DeVane, Ph.D. candidate and former Zen priest
Fear sounds an alarm that our needs are in jeopardy and demand our attention or protection. So in all the AI doom fear, I ask myself: Whose needs are in jeopardy? Why are they sounding an alarm? Ninety-nine percent of us in this conversation are fearful about less access to meeting our basic needs, for which employment is central, at a time when we’re already feeling pinched with increasing cost of living. But leaders in tech are telling us to worry about something much bigger: the end of civilization. I don’t doubt that some of them mean it sincerely, but sincerity and social structure aren’t the same thing. The risk is structurally significant enough that we can’t trust it to be managed by a handful of companies run by people who can meet their own basic needs at orders of magnitude beyond anyone else, even if the leaders sound sincere.
My biggest fear is that without legal oversight, the voracious drive for profit in the tech industry will continue to come at the expense of human welfare. I see an unregulated tech industry as a road toward further erosion of our sense of social responsibility to one another, which is being obscured by all the big emotions around technological change. We need political action based on collaboration. We need to collectively calm ourselves first, though, before we can be productive.
The Founder: Slowing development won’t solve anything
Will Wilson, CEO of code-testing platform Antithesis
Even if we figure out a way to “align” AI systems so they reliably do what we want, I have a hard time imagining a scenario in which sharing the Earth with a superintelligent AI ends well for us. Anything that renders human beings economically and militarily unnecessary will reduce the need for governments to treat their citizens well. When you combine that with the incredible ability to impose surveillance and control that this technology provides, I don’t think any society will remain democratic or free once powerful AGI systems are commonplace. Slowing the frontier may buy time, but “pacing” alone can’t solve that underlying problem.
I think recursive self-improvement of AI systems in the near future is relatively unlikely, but the situation remains extremely dangerous for more prosaic political and economic reasons. My greatest fear is not necessarily a rogue AI suddenly ending humanity; it is that powerful, nominally aligned systems gradually make human beings irrelevant and enable a durable form of totalitarian control.