The new SimpleBench examination uses more than 200 multiple-choice questions to evaluate skills such as spatio-temporal reasoning, linguistic adversarial robustness, and social intelligence. While leading systems frequently surpass people on standard text evaluations, every tested machine fell short of the human baseline. Even the highest-scoring model, Claude Fable, could not surpass the average performance achieved by non-specialist individuals. Benchmark creators emphasized that approximate reasoning and memorization are still unable to consistently navigate basic trick questions.
Key points
Frontier language models continue to fail at matching non-specialized human capabilities…: Frontier language models continue to fail at matching non-specialized human capabilities on foundational reasoning benchmarks.
Advanced models rely heavily on memorized knowledge and approximate retrieval that prove…: Advanced models rely heavily on memorized knowledge and approximate retrieval that prove insufficient for basic common-sense problems.
Superior performance on traditional text benchmarks does not guarantee that artificial in…: Superior performance on traditional text benchmarks does not guarantee that artificial intelligence models possess robust linguistic or spatio-temporal reasoning.
Sources checked by our analysis pipeline: 1
We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals
with
unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200
questions covering spatio-temporal reasoning, social intelligence, and what we call
linguistic adversarial robustness (or trick questions). For the vast majority of text-based
benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human
performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on
our small sample of nine participants, outperforming every tested LLM, including
today's top model, Claude Fable, which scored 81.9%. While we expect model performance to improve over time, the
results of SimpleBench confirm that the memorized knowledge, and approximate reasoning
retrieval, utilized by frontier LLMs is not always enough to answer basic questions just
yet.
For a deeper dive into our results and our methods, check out the full technical report here.
Use all of these models on the Simple Bench app - LMcouncil.ai
Geneva delegates concluded a week of late-night negotiations by establishing a two-year window to propose regulations for lethal autonomous systems, leaving the final measures potentially optional. The concession was driven by major military powers that resisted rapid, mandatory prohibitions pushed by humanitarian leaders. This delay leaves rapidly evolving artificial intelligence weapons governed without formal international guardrails amid ongoing global conflicts.
Delegates at the UN Review Conference on Conventional Weapons in Geneva failed to establish a negotiating mandate for legally binding rules on artificial intelligence in warfare due to Russian objections. The setback leaves automated targeting systems unrestrained by new international statutes despite their active deployment in modern conflicts. With strict consensus rules stalling diplomacy for decades, supporters may now be forced to seek outside diplomatic channels to enact restrictions.
CoreWeave's aggressive expansion across 52 worldwide datacentres has saddled the infrastructure provider with $47 billion in debt to shield partners like OpenAI from direct building costs. This highly leveraged model faces intense pressure from impending debt maturities and deep quarterly losses. Furthermore, sudden leadership changes at Nvidia threaten to destabilize the circular financing arrangements that sustain the entire operation.