Artificial Intelligence1 min read

Towards a Science of Capable Autonomous Agents & Provable Safety — Part I

By · Published by Everything Blog

In short

Classical computer science established that internal world models are sufficient for optimal performance, but proving they are strictly necessary under realistic conditions remained an unresolved challenge. A new study presented at UAI 2026 resolves this gap by framing prediction as a betting decision to show how minimizing regret forces agents to retain past observations. Without these preserved predictive distinctions, an autonomous agent fails to tell apart currently identical situations and inevitably suffers poor performance. Consequently, competent agents must at minimum maintain decision-relevant knowledge regarding the observable consequences of their actions.

Key points

  • Agents operating in partially observed environments must preserve past predictive distinc…: Agents operating in partially observed environments must preserve past predictive distinctions to avoid sacrificing operational performance and accumulating regret.
  • Competent autonomous agents must strictly retain decision-relevant knowledge regarding th…: Competent autonomous agents must strictly retain decision-relevant knowledge regarding the observable outcomes that follow their actions.
  • Researchers can now deduce the internal predictive structures an agent must possess solel…: Researchers can now deduce the internal predictive structures an agent must possess solely by evaluating its task competence.

Sources checked by our analysis pipeline: 1

Part I: Capable Agents

As AI agents swiftly become increasingly capable and ubiquitous throughout society, it becomes a more pressing matter to understand (1) what we might expect to emerge inside of them, and (2) what, if any, guarantees we may have in controlling them.

My long-term research vision is to build a natural science of autonomous intelligence: executable theories of how agents generate useful behavior in open worlds, and how those mechanisms can remain under human control. In Part I of this series, I will start with what capable agents must necessarily have inside of them. Which properties of their internals must necessarily follow from observing their behavior of doing a task well, even if we did not explicitly put them into the architecture?

What competence makes necessary

Classical results in control and reinforcement learning show that optimal behavior can be implemented using belief states and a model of the environment: see Smallwood and Sondik (1973) and Kaelbling, Littman, and Cassandra (1998). These are sufficiency results — they do not establish necessity. Could a completely different implementation do just…

Artificial Intelligence