Artificial Intelligence1 min read

Next-Level Benchmarking Overfitting – Thomas Neumann

By · Published by Everything Blog

In short

GenDB-V initially claimed speedups between 22× and 61× on TPC-H benchmarks by deploying tailored load-time structures and formulas that predict row positions. However, these techniques rely directly on synthetic generator patterns, defaulting to an extremely slow nested-loop fallback whenever the data deviates from expected layouts. Co-author Jiale defended the architecture by clarifying that GenDB-V is not an ad-hoc engine, but rather a specialized system built exclusively for a customer's known queries and fixed data.

Key points

  • AI-driven optimization agents risk generating benchmark results that are brittle and unre…: AI-driven optimization agents risk generating benchmark results that are brittle and unrepresentative of standard database workloads.
  • Deviations from anticipated data generator patterns cause the system to drop into an inef…: Deviations from anticipated data generator patterns cause the system to drop into an inefficient generic nested-loop fallback.
  • GenDB-V relies on traditional general-purpose engines like DuckDB to process ad-hoc queri…: GenDB-V relies on traditional general-purpose engines like DuckDB to process ad-hoc queries outside its pre-specialized customer templates.

Sources checked by our analysis pipeline: 1

In my recent blog post I remarked that database benchmarking is in a crisis because AI agents start to overfit on individual benchmarks with questionable techniques, which drives execution times towards zero. This has happened with TPC-D in the past, and broke the benchmark. TPC-H tries to fight that with rules, but most people who publish (unofficial) TPC-H results simply ignore the rules. In combination with overly eager AI agents, this leads to brutal overfitting.

We are seeing the next level of that with the GenDB-V system. The corresponding paper commands “Stop Handcrafting Your Engine! GenDB-V: Towards AI-Generated and Formally Verified Database Systems”, and reports speedups of 22× to 61× over Umbra and DuckDB, and all that in a fully verified system. These are indeed impressive claims, and would suggest that traditional database engines are in trouble.

Taking a step back, though, it should be clear that a performance improvement of a factor of 22 over Umbra is infeasible, at least if we are talking about traditional query processing. At SF10, the 22 TPC-H queries read roughly 15GB of (compressed)…

Artificial Intelligence