Artificial Intelligence1 min read

Astra 6.2 and Sol 6.2 flop at Simple Bench

By · Published by Everything Blog

In short

The new SimpleBench examination uses more than 200 multiple-choice questions to evaluate skills such as spatio-temporal reasoning, linguistic adversarial robustness, and social intelligence. While leading systems frequently surpass people on standard text evaluations, every tested machine fell short of the human baseline. Even the highest-scoring model, Claude Fable, could not surpass the average performance achieved by non-specialist individuals. Benchmark creators emphasized that approximate reasoning and memorization are still unable to consistently navigate basic trick questions.

Key points

  • Frontier language models continue to fail at matching non-specialized human capabilities…: Frontier language models continue to fail at matching non-specialized human capabilities on foundational reasoning benchmarks.
  • Advanced models rely heavily on memorized knowledge and approximate retrieval that prove…: Advanced models rely heavily on memorized knowledge and approximate retrieval that prove insufficient for basic common-sense problems.
  • Superior performance on traditional text benchmarks does not guarantee that artificial in…: Superior performance on traditional text benchmarks does not guarantee that artificial intelligence models possess robust linguistic or spatio-temporal reasoning.

Sources checked by our analysis pipeline: 1

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals

with

unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200

questions covering spatio-temporal reasoning, social intelligence, and what we call

linguistic adversarial robustness (or trick questions). For the vast majority of text-based

benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human

performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on

our small sample of nine participants, outperforming every tested LLM, including

today's top model, Claude Fable, which scored 81.9%. While we expect model performance to improve over time, the

results of SimpleBench confirm that the memorized knowledge, and approximate reasoning

retrieval, utilized by frontier LLMs is not always enough to answer basic questions just

yet.

For a deeper dive into our results and our methods, check out the full technical report here.

Use all of these models on the Simple Bench app - LMcouncil.ai

Rank | Model | Score (AVG@5) | Organization |

---|

Artificial Intelligence