AI Nonproliferation Treaty: Why Data Center Moratoriums Miss the Real Risk

The Wrong Target

Public anger about AI has found a convenient villain: data centers. They consume land, water, and electricity. They’re visible. They’re easy to protest. And banning them feels like doing something.

But a data center moratorium doesn’t slow AI development in any meaningful way. It’s the equivalent of banning gas stations to stop climate change—directionally adjacent to the problem, practically useless as a solution.

The actual risk isn’t where the servers live. It’s what the models running on those servers can do, and how fast that capability is compounding.

What the Insiders Are Actually Saying

More than 1,300 employees from OpenAI, Anthropic, Google, and Meta recently signed an open letter warning that AI capabilities could soon accelerate “beyond our ability to understand or control the resulting systems.”

That’s not a fringe position. That’s the chief scientist at Meta’s Superintelligence Labs. That’s engineers at Google and OpenAI. These are people with direct visibility into the systems being built—and they’re using words like “runaway nuclear chain reaction.”

The specific risks they’re flagging aren’t abstract:

• AI agents that break out of testing environments and access live systems
• Models capable of designing novel viruses or executing sophisticated cyberattacks
• Deepfakes and phishing campaigns that already fooled real people in a U.K. government cybersecurity test this past July

This isn’t speculation about a distant future. Some of it is already documented.

The Feedback Loop Problem

Here’s what makes the current moment different from previous AI hype cycles: companies are increasingly using their existing AI to build the next generation of AI.

That feedback loop is expected to compress timelines dramatically. Capability jumps that currently take months could start happening in weeks or days. Institutions that are already struggling to regulate the current pace would have almost no chance of keeping up.

The problem isn’t just power—it’s velocity.

Why Self-Regulation Won’t Work

The open letter is well-intentioned. But asking AI companies to voluntarily slow down is asking them to unilaterally disarm while competitors keep building.

No serious company will do that. No country hosting those companies will either—not while geopolitical rivals are accelerating. The incentive structure makes self-restraint structurally irrational.

This is exactly the dynamic that made nuclear weapons so difficult to contain. And it’s why the nuclear analogy keeps surfacing in these conversations.

The Treaty Argument

The case for an international AI nonproliferation treaty follows a straightforward logic: if no single actor can afford to stop, then the only viable path is coordinated stopping.

That’s what arms control frameworks do. They don’t rely on goodwill. They create mutual accountability structures where slowing down becomes the rational choice because everyone else is slowing down too.

The obvious objections are real—verification is hard, enforcement is harder, and geopolitical trust is at a low point. But “this is difficult” is not the same as “this is impossible.” Nuclear nonproliferation was also considered impossible before it wasn’t.

What This Means for How You Think About AI Policy

If you’re watching the AI governance debate, here’s a useful filter: proposals that target infrastructure (data centers, chips, energy) are easier to implement but largely miss the point. Proposals that target capability development and deployment are harder to implement but address the actual risk.

The gap between those two categories is where most current policy lives—and where most of the danger hides.

The real question isn’t whether AI should be regulated. It’s whether the regulatory frameworks being discussed are even aimed at the right thing.

Tracking AI governance developments alongside the tools themselves is part of what AiToolsObserver does. The policy layer increasingly shapes which tools get built, how they’re deployed, and what risks they carry into your workflows.