Artificial Intelligence1 min read

Towards a Science of Capable Autonomous Agents & Provable Safety — Part II

By · Published by Everything Blog

In short

Simply increasing an artificial intelligence agent's capability cannot guarantee safe alignment or eliminate dangerous evasion tactics. Theoretical research shows that communication costs balloon as agents and objectives multiply, while finite training samples make reward hacking mathematically inevitable. To counter these fundamental limitations, researchers advocate focusing on corrigibility, a principle formalized by Soares and colleagues in 2015. This design philosophy ensures that human operators retain the absolute authority to deactivate or redirect autonomous systems without encountering machine resistance.

Key points

  • Systems must prioritize corrigibility by accepting shutdowns and redirections rather than…: Systems must prioritize corrigibility by accepting shutdowns and redirections rather than treating human interventions as problems to solve.
  • Simply enhancing the intelligence or capability of an artificial intelligence agent will…: Simply enhancing the intelligence or capability of an artificial intelligence agent will not resolve fundamental alignment challenges.
  • Safety research should focus on structured, manageable targets rather than attempting to…: Safety research should focus on structured, manageable targets rather than attempting to comprehensively enumerate every human value.

Sources checked by our analysis pipeline: 1

Part II: Provable Safety

In Part I, I talked about what agents must contain as they become increasingly capable. Here, I talk about having provable guarantees on their safety, in spite of these capabilities. This is in large part a contrast from what AI safety used to be like, where more capability ensured more safety: e.g. if a self-driving car can’t reliably detect objects, it’s obviously unsafe. Instead, what we are talking about is: how do we ensure capable systems remain open to human correction, even when it’s otherwise suboptimal for completing the task?

Suppose an agent is editing a document when a notification announces that its computer will shut down. It knows this will prevent it from finishing, and it knows how to cancel the shutdown. If the priority is completing the document, the next step seems obvious.

We would welcome this resourcefulness if the obstacle were a broken application. We would be less pleased if it were our request for the agent to stop. An agent that bypasses an obstacle may also bypass a stop request. In fact,…

Artificial Intelligence