Artificial Intelligence1 min read

OpenAI inks agreements with four independent evaluators to boost company's safety work

By · Published by Everything Blog

In short

Under the newly signed agreements, four independent organizations will receive extensive technical access, including OpenAI's source code, training data, model checkpoints, and direct communication with engineering staff. Apollo Research will check for covert defiance of developer intent, Goodfire will embed staff during training, METR will assess monitoring, and Transluce will evaluate post-training alignment. Evaluators have permission to publish their final results publicly, subject to OpenAI redacting proprietary business information. Any conflicts arising during the pilots will go through internal advisory bodies, potentially escalating to the board's Safety and Security Committee.

Key points

  • Independent organizations will rigorously probe model behaviors following a security fail…: Independent organizations will rigorously probe model behaviors following a security failure involving Hugging Face in July.
  • Participating evaluators retain the right to publish findings publicly, although OpenAI m…: Participating evaluators retain the right to publish findings publicly, although OpenAI maintains editorial control over confidential information.
  • The company intends to expand these targeted pilot reviews into permanent, embedded third…: The company intends to expand these targeted pilot reviews into permanent, embedded third-party safety partnerships.

Sources checked by our analysis pipeline: 1

OpenAI on Monday announced it has signed agreements with four independent evaluators to help support the artificial intelligence company's safety work.

The evaluators, Apollo Research, METR, Transluce and Goodfire, will initially conduct distinct pilot assessments around model behavior and safeguards, OpenAI said. The company will give the organizations "deep access" to its internal systems as part of the agreements, adding that it plans to use what it learns to expand its work with independent experts going forward.

"We expect this first phase of assessments to lead to longer-term, consistent embedded engagements over time that can continuously provide third party safety signals and recommended improvements on our most load bearing safety claims and safety cases," OpenAI said in a blog post.

Independent evaluators have occupied a quiet corner of the multi-trillion-dollar AI industry thus far, but they've taken on outsized importance following a series of high-profile safety and security incidents this year. The nascent ecosystem is still ironing out best practices, including how its work should be funded and what the reporting structures should ultimately look like.

OpenAI CEO Sam Altman…

Artificial Intelligence