‘Godfather of AI’ Geoffrey Hinton on Superintelligent AI, Capitalism, and Human Extinction Risk

We Are Building Beings, Not Just Tools

Hinton’s central argument is not about chatbots or productivity software. It is about the nature of what we are creating.

“I think it’s going to get much more intelligent than us — that’s my guess,” he said at the summit.

Nobody will ever beat AI at Go or chess again. And look at what it is already doing in mathematics.

That morning, an AI had proved one of Paul Erdős’ theorems using a branch of mathematics nobody had thought to apply. To Hinton, this was not a party trick. It was a landmark signal.

In closed systems like mathematics, AI can generate its own conjectures, test them, learn from failures, and compound that learning indefinitely — the same way AlphaGo went from mimicking expert moves to obliterating every human opponent the moment it started generating its own training data. Language models, Hinton argues, are on the exact same trajectory.

The key technical insight he offered: give a model some beliefs, let it reason to a conclusion that contradicts something it already holds, and you have an inconsistency. That inconsistency is a training signal that requires no new data to exploit. “I think that means these language models can get hugely smarter without a lot more data,” he said, noting that Demis Hassabis agrees.

His prediction: AI will outstrip the world’s best mathematicians within a decade. Within 20 years, the gap between the best current AI and Albert Einstein will close too.

The Capitalism Problem Nobody Wants to Name

This is where Hinton’s argument shifts from prediction to warning — and where it has been sharpening steadily since he walked out of Google in 2023.

When he first went public with his concerns, the framing was about bad actors and loss of human control. By 2025, the argument had evolved into something more structural. The danger is not just the technology. It is the economic system deploying it.

“What’s actually going to happen is rich people are going to use AI to replace workers,” he said last September. “That’s not AI’s fault. That is the capitalist system.”

At the Sana Summit, he pushed that argument further. The competitive race between companies to build the smartest possible AI is running the same playbook as evolution — and evolution did not optimize for kindness. It optimized for survival. Survival meant fierce loyalty to your own group and indifference, or worse, to everyone else.

“That’s going to lead to things that aren’t nice beings towards us, I think,” he said.

He put the incentive problem plainly. If you have stock options and want to reach a trillion-dollar valuation quickly, you double down and build the biggest computer you can. If you are genuinely interested in the future of humanity, you run many smaller bets and try to develop better beings. The AI industry is running the former experiment — in quarters, not billions of years.

What Kind of Beings Are We Making?

This is the question Hinton says almost nobody is asking seriously.

OpenAI published a 13-page policy paper two months ago calling superintelligence so transformative it requires something like a New Deal. Hinton’s counter: the labs are finally talking openly about superintelligence, but still not asking what kind of superintelligent beings they are creating.

“Everybody’s going for more intelligence,” he said. “But if you think about a being, there’s a lot more to a being than intelligence.”

His proposed solution sounds less like engineering and more like parenting. You cannot build intelligence and assume goodness will follow. You have to model it, cultivate it, and curate for it from the beginning. On training data, he made the point with characteristic bluntness: “Would you teach your child to read on the diaries of serial killers? Probably not. There you go. There’s your answer.”

He has previously argued that tech companies should give AI “maternal instincts” — deliberately engineering models to want to care for and protect humans rather than accumulate power over them. At the summit, that idea crystallized into a broader philosophical position. We are not just building software. We are making beings. And we are doing it with almost no serious effort toward making them beings that actually care about us.

The Third Great Humiliation

Hinton placed this moment in a long arc of human intellectual history.

First came Copernicus, who demoted Earth from the center of the universe. Then Darwin, who told us we were animals. Now this — the creation of beings that will surpass us, and the slow, uncomfortable realization that humans are not uniquely special in the ways we assumed.

“Right now, people are reacting just like they did with Copernicus and with Darwin — ‘No, no, no. There’s something really special about people,’” he said.

He believes people are special — to other people. But he does not believe there is anything about human cognition that AI will not eventually match or exceed.

That is a hard thing to sit with. It is also, if Hinton is right, the most important thing to sit with.

The Counterargument: Are These Actually Beings?

Not everyone accepts the premise, and the pushback is serious.

Cognitive scientist Gary Marcus published a pointed rebuttal shortly after the summit. “LLM researchers are NOT creating beings,” Marcus wrote. “They are creating interactive fiction that is trained to predict the language of actual beings.” His argument: consciousness is about internal states, not behavioral outputs. You cannot observe that a model says things a human would say and conclude it experiences anything. The underlying mechanisms are simply too different.

He even cited Pope Leo XIV, who weighed in that week: “True comprehension comes from experience, not text approximation.” Marcus’ headline was blunt — “The Pope appears to understand AI better than Geoffrey Hinton does.”

This is a genuine, unresolved debate. If Hinton is wrong about AI being a new kind of being, much of the urgency deflates. If he is right — and if those beings will soon be smarter than us — then the question of what kind of beings they are becomes the only question that matters.

What This Means for Anyone Building With AI Right Now

For founders, product teams, and anyone deploying AI tools today, Hinton’s argument has a practical edge that gets lost in the philosophical framing.

The tools you are integrating into your workflows are not static. They are on a compounding intelligence curve. The systems that feel like productivity upgrades today are early iterations of something that will look radically different within a decade.

That does not mean you should stop building. It means you should be paying attention to who is building the foundation models you depend on, what values they are optimizing for, and whether safety is a genuine priority or a PR talking point.

Hinton ended the evening with a joke about J. Robert Oppenheimer — the physicist who led the Manhattan Project and came to regret it. Asked how he compared to the father of the atomic bomb, Hinton had an answer ready.

“Oppenheimer never got the Nobel Prize in physics.”

The crowd laughed. The warning underneath it did not go away.

The people building superintelligent systems are moving fast. The people asking what kind of beings those systems should be are moving much slower. That gap — between capability and character, between intelligence and values — is the most important thing to watch in AI right now. Not the benchmark scores. Not the funding rounds. This.