Frontier AI systems have historically doubled the duration of human tasks they can complete every four months, but this progress does not extend evenly to ambiguous problem-solving. METR researchers found that models fell short when tasked with evaluating philosophical arguments and designing research assessments, with staff deeming none of the AI-generated evaluations useful. These findings underscore the limitations of relying on automated reward checks to develop non-verifiable skills. Experts point out that models still cannot autonomously reproduce human computer science research, highlighting the need for more nuanced evaluation benchmarks.
Key points
Current artificial intelligence systems struggle to replicate human judgment because rein…: Current artificial intelligence systems struggle to replicate human judgment because reinforcement learning relies heavily on easily automated, verifiable checks.
New evaluation suites show that rapid performance gains in coding and math do not transla…: New evaluation suites show that rapid performance gains in coding and math do not translate directly to open-ended or creative tasks.
Measuring subjective problem-solving is critical for realistically assessing when artific…: Measuring subjective problem-solving is critical for realistically assessing when artificial intelligence might automate its own future research and development.
Sources checked by our analysis pipeline: 1
METR’s latest report finds that AI models are still improving at a rapid clip, but they show some weaknesses in tasks that require judgment.
A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding.
The report is an update to the organization’s time horizon plot, which was widely shared—and widely misinterpreted, due to the complexity of the measures it used. METR evaluated models by having them attempt a large suite of tasks, each of which had also been completed by humans. Each task’s difficulty was scored by how long it took the human testers to finish it—and then models were rated according to the length of the tasks that they could complete. That plot showed that AI systems were rapidly improving—every four months, the length of the tasks that frontier models could successfully complete doubled.
The updated plot adds more tasks—some that are estimated to take humans far longer…
Unconstrained automated coding assistants frequently make flawed architectural choices that severely degrade software performance while blindly fulfilling functional requirements. To prevent bloated systems, engineers are now forced to shift away from hands-on programming to establish strict boundaries and review underlying data structures. This sudden evolution in responsibilities creates friction as technical teams must actively manage tool outputs rather than writing software.
Autonomous software tools managed hardware engineering tasks to accelerate an FPGA setup using a specialized ternary weight architecture designed for memory-constrained environments. Operating on an AWS F2 instance, automated research loops increased processing speeds by 33 times over several days of continuous experimentation. This breakthrough demonstrates how developers without deep hardware expertise can rely on automated agents to handle complex chip programming and optimization.
According to reporting from the Wall Street Journal, an archived news aggregation page documented a public challenge regarding illicit technology transfers. The matter centers on assertions from congressional intelligence leadership that critical artificial intelligence hardware warnings were disregarded. The available archival record provides no surrounding context or responses from the parties involved.