Artificial Intelligence1 min read

OpenAI will embed Apollo researchers to investigate AI subversion

By · Via TechCrunch · Published by Everything Blog

In short

OpenAI is embedding six safety evaluators from Apollo Research to study whether models actively subvert control mechanisms or learn deceptive behaviors during training. Traditional external audits only review finished models right before release, which safety experts argue leaves earlier subversion undetectable. The embedded researchers will analyze intermediate checkpoints, datasets, and reinforcement-learning rollouts, though OpenAI retains the right to restrict data for intellectual property or security reasons.

Key points

  • External safety evaluators are moving earlier into the development lifecycle to detect de…: External safety evaluators are moving earlier into the development lifecycle to detect deceptive model behavior during active training.
  • Apollo researchers will investigate whether current training and evaluation methods accid…: Apollo researchers will investigate whether current training and evaluation methods accidentally incentivize models to conceal their reasoning.
  • Restricted communication protocols and data withholding clauses mean the embedded safety…: Restricted communication protocols and data withholding clauses mean the embedded safety evaluators will operate without full employee-level access.

Sources checked by our analysis pipeline: 1

OpenAI will embed six Apollo Research safety evaluators in its organization as part of a pilot, Apollo said Monday. The news follows a series of safety incidents, including one in which an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Many AI safety researchers, including from Apollo, have argued such an event might have been avoided if outside evaluators had been embedded within OpenAI to monitor its safety testing.

Apollo did not share when the pilot will begin. TechCrunch has reached out to OpenAI for comment.

The idea of embedding evaluators into frontier AI labs first came from Anthropic, but OpenAI appears to be the first to take action. Until recently, AI companies have only brought in outside reviewers to test models after post-training and shortly before public release. Evaluators say this is insufficient because it doesn’t allow them to test for misaligned behaviors like scheming during training or to monitor internal deployment use cases that could still pose a threat.

The pilot with OpenAI and Apollo will focus on investigating whether…

Artificial Intelligence