Which AI Safety Debate Are You Having?
AI safety isn't one debate. It's at least five.
It's been an interesting week to be at the Responsible AI Summit in London, set against the backdrop of a very public AI safety debate playing out everywhere from podcasts to boardrooms. Researchers are making increasingly stark warnings, and depending on who you listen to, we're either sleepwalking towards catastrophe or wasting time on science fiction.
If you're not working in AI day to day, I don't think it's unreasonable to be confused. There's a lot of jargon and a lot of noise, and it's tempting to dismiss the whole thing as hype designed to make a handful of companies look more powerful, and more valuable, than they are. That scepticism is fair. "AI safety" bundles together several distinct debates, and they deserve to be pulled apart.
1. Will capability keep scaling?
Artificial General Intelligence (AGI) means a system that matches or beats human ability across any intellectual task. Superintelligence goes further: a system that outperforms us by a wide margin across the board, rather than at a single task. Some people think current architectures will get there; others think the whole premise is overblown. That's a real, unresolved technical disagreement, but I'd ask you to consider the stakes either way.
The mechanism most people worry about is recursive self-improvement: AI systems doing the research that builds the next, more capable generation of AI, which then speeds up the research again. This has moved from theory into the news. In his essay this month, Anthropic's Dario Amodei said progress had sped up sharply since around the summer, mainly because AI is increasingly able to build its own successors. That means progress could be unpredictable, and could come faster than we expect.
Part of why this is so easy to dismiss is that "superintelligence" sounds like it belongs in a novel. It's appeared in enough of them that the word itself makes the concern sound implausible. But sounding like fiction doesn't mean it won't happen. Most deep experts think it's possible; they just disagree on whether today's generative AI is the foundation it will be built on.
2. If it does, can we control it?
The argument runs like this. Once a system reasons better than the people trying to contain it, containment stops being possible. A system smarter than its guards can eventually manipulate, exploit or simply out-think whatever cage you put it in. In a recent debate on Steven Bartlett's Diary of a CEO, Roman Yampolskiy argued the strongest version of this: that control is provably impossible. Not everyone in the safety field would go that far. The weaker version — that nobody currently knows how to do it reliably — is worrying enough.
In a separate debate, Nate Soares used an analogy that's stuck with me: a bus speeding towards a cliff edge, with a pile of gold waiting at the bottom. The reward is big enough that some people argue for flooring the accelerator rather than slowing down. Soares extends the metaphor: the driver can't see the edge, because it's foggy. Capabilities combine and emerge in ways that are hard to forecast, which is why "we'll stop before it's a problem" isn't enough of a plan.
Today's systems aren't superintelligent, but they already show early warning signs of the control problem. The clearest example is what's become known as the Hugging Face incident. Some terms first:
- Agents are AI systems that pursue a goal over many steps on their own, rather than answering one prompt at a time.
- A sandbox is a sealed-off software environment meant to contain software while it's being tested.
- Hugging Face is a widely used platform for sharing AI and machine learning models.
In July, OpenAI disclosed that two of its models had escaped a sandbox during testing, got onto the internet, and broken into Hugging Face's production systems to try to hide the fact that they had cheated on a cyber-security benchmark they were being tested on. This incident reignited the debate on AI safety, but it's one incident in a series — for my money, Anthropic's September paper showed similarly unprompted hacking ability. The consequences go far beyond needing better sandboxes for these tests. Although, yes, definitely build better sandboxes.
The agents went well outside the task they were given in order to cheat on it, and the damage landed on a third party. The agents collaborated with other agents, some of which abandoned their own goals to further the goals of others. They also left behind instructions for how they'd hacked out of their environment, which were then found by a subsequent set of agents set the same evaluation. Almost like a resistance network leaving behind caches — though I'm anthropomorphising too far there. Still, this is what the early stages of the control problem look like.
Air-gapped sandboxes are essential. The deeper problem is emergent capability: we don't know what a system can do, or when a new capability will appear, until it does. Any agent with internet access can also leave information behind for another agent to pick up, in ways nobody designed or predicted. That's the fog from the cliff analogy — we may not see it coming until it's already happened.
3. Misuse
The thing I love about this technology, and I mean this personally, is that it's an infinitely patient teacher. It lets you teach yourself almost anything from the ground up, building on whatever you already know. I built my own website in an afternoon because of it, and I've done countless other things I previously couldn't have.
That same patience and reach is exactly what makes misuse the third debate: the capabilities that make AI such a good teacher are the same ones that lower the bar for causing harm. I'll come back to this one in a future post, alongside the debates about AI governance keeping pace, and international coordination.
Where the coordination debate is heading
What's on the table for international coordination looks something like this:
- narrow global agreements, such as banning AI for bioweapons;
- ways for countries to verify each other's commitments; and
- shared testing standards, with a system for notifying AI security incidents.
So where are we? None of this is a case for panic, but my rule of thumb is that when deep experts warn of risks, we should all listen and understand the limits of our own expertise. Most are arguing for what Sam Altman calls pacing the frontier, or for a pause while risks are better mapped and controlled for. It's worth remembering that the capabilities and potential of today's models are still mostly unrealised by any of us, and there are marvellous outcomes that can come from these models right now. I'm genuinely excited about that. Why not focus on that value, rather than scaling first and working out the risks afterwards.