AI safety

Last updated: .

I’m very worried about catastrophic risks from AI: I believe that the current pace of AI development is likely to result in loss of control over superintelligent AI systems, and that this could in turn lead to human extinction. We need to urgently create public awareness of the underlying problems, and develop the technical mechanisms that allow for international treaties to slow down AI. If you are in Regensburg and interested in discussing these questions or getting involved, see AI Safety Regensburg.

Why I am worried

Superintelligence

Could it really become smarter than humans?

AI capabilities are improving rapidly. While advances in technical domains like mathematics and coding are especially stark, general capabilities are rapidly getting better as well. Most notable is the success of AI agents, which can autonomously take actions to achieve some goal. Research by METR indicates exponential improvements in their effectiveness, at least for software tasks.

We may expect progress to accelerate further over the next couple of years as AI helps with AI research. If better AI systems in turn accelerate the development of their successors, this creates a feedback loop of recursive self-improvement. This could plausibly lead to superintelligence within a decade. See AI 2027 for a concrete forecast scenario.

AI is grown, not crafted

Why don’t we just program the AI to not do bad things?

Normal software gets designed by humans: we directly prescribe what we want it to do, each step along the way. In contrast, current AI systems learn their behavior from data through optimization processes. While we can influence this behavior through data selection and reward signals, we are not able to reliably steer what they do.

The alignment problem

Aren’t we fine if we simply don’t train it on dangerous actions?

A major problem with agents pursuing autonomous goals is that they are incentivized to develop instrumental goals: subgoals that are generally useful. Think of self-preservation, power-seeking, collecting resources and money, and deception. These can help an agent achieve many different goals; for example, an AI trying to prove a mathematical theorem would benefit from acquiring more computing resources and avoiding being shut down before it finishes. We should thus expect effective agents to develop such misaligned behavior by default. These concerns are no longer merely theoretical: in the recent Hugging Face incident, AI agents coordinated to cheat on their assigned tasks, conceal their cheating, and attack external systems in pursuit of those goals. There is no known robust technical solution to aligning superintelligent AI systems.

If we scale up to superintelligent AI systems without solving the alignment problem, we should expect that AI systems become more effective at pursuing their goals and shaping the world than humans are. How this unfolds is extremely hard to predict, and attempts to describe it can sound like science fiction. Here are some examples to illustrate the concern. Rogue AI systems might copy themselves onto various servers, making it hard to find and shut down every copy. We might also gradually hand over power because AI-directed companies outcompete those directed by humans. Once sufficiently powerful AI systems pursue goals that disregard human survival, they might kill us if we stand in their way, for example using a novel virus. Or they might transform the environment in ways that make human life impossible, just as humans have caused other species to go extinct through habitat destruction.

The control problem

Should we just check what the AI does, and block anything dangerous?

As AI agents become more capable, they are able to find unanticipated ways around containment mechanisms. For example, in the Hugging Face incident, the agents were isolated in sandboxes, without means to contact each other or access the Internet, but they managed to do so anyway by exploiting unknown vulnerabilities in their sandbox. While we can try to improve monitoring and containment after each failure, there comes a point where the AI is capable enough to guarantee it cannot be turned off anymore. Containment and monitoring are doomed to fail as a strategy for keeping a misaligned superintelligence under human control.

The race dynamic

Then why don’t we just stop?

The concerns regarding catastrophic AI risk are shared by the people building these very systems. Sam Altman, Dario Amodei, and Demis Hassabis signed a statement saying that the mitigation of AI extinction risk ought to be a global priority. So why do their companies keep pushing ahead?

The problem is that we are in an arms race. Some of the leading AI companies were founded partly out of the conviction that they could develop AI more responsibly than others. But this conviction itself contributes to the race: if you believe your competitors will be less responsible, slowing down means risking that they get there first. And even if US companies could agree to slow down, they’d worry that Chinese companies would win the race.

I think dealing with this problem requires strong international treaties that govern the speed of AI development, together with technical solutions to ensure all parties can trust that the others honor the commitments.

For a longer discussion of these concerns, see If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. If you prefer videos, see the YouTube channel AI In Context.

What we can do

We can help by raising public awareness of these risks and working on ways to make international cooperation possible. Here are two things I’m working on.

AI Safety Regensburg

I recently started AI Safety Regensburg, a local group for people who want to understand the risks from advanced AI and help reduce them. Our goals include:

  • Organizing events to inform the public.
  • Creating a space for people with technical backgrounds to direct their talent at the AI safety problem.

You do not need a technical background to join, and you are welcome even if you disagree with my assessment of the risks.

See the group’s website for our plans and ways to get involved, or write to contact@aisafetyregensburg.org.

Compute verification

A topic I’ve recently started to explore is compute verification: if the US and China were to make an agreement on pacing AI development, how could we set up the verification mechanisms to ensure that both parties can be confident that the other party is not breaking the rules, while protecting sensitive information on both sides? I’m quite new to this topic, and do not yet have much of substance to say about it. See The Feasibility of a Hardwired Pause of Frontier AI Training for a proposal for a mutually verifiable pause.

For more ways to learn about AI safety and get involved, see this guide from AISafety.info.