Technology1 min read

Lakegres: Vector search on existing Open/Hudi/Iceberg tables

By · Published by Everything Blog

In short

The native vector indexing feature in Lakegres functions entirely as derived metadata, meaning it can be created or dropped without modifying the original data files. Updates to the index happen alongside the regular commits that alter the tables, organizing vectors into clusters using compact quantized codes. Benchmark tests across 247 million vectors demonstrated median warm query responses of 63.5 milliseconds, outperforming Databricks AI Search warm median latencies by roughly tenfold. This technology builds on Onehouse tools that parse media files into embeddings stored inside open table architectures.

Key points

  • Organizations can query vector data through a Postgres endpoint without exporting lake-sc…: Organizations can query vector data through a Postgres endpoint without exporting lake-scale records into external systems.
  • Vector indexes operate as derived metadata that can be added or deleted without rewriting…: Vector indexes operate as derived metadata that can be added or deleted without rewriting the underlying table files.
  • Index updates are processed seamlessly within the exact same commits that modify the prim…: Index updates are processed seamlessly within the exact same commits that modify the primary lakehouse data.

Sources checked by our analysis pipeline: 1

Talk with our engineers by joining the new Onehouse community Slack!

TL;DR

- Problem: Vector search is a standard feature of application databases. Lake and warehouse data still has to be copied into a separate search system before an agent can search it.

- Why now: Agents query all of a company’s data. Most of that data is stored in open tables that are 100 to 1,000 times larger than any application database.

- What we built: Lakegres now provides vector search through an index stored inside existing and new Hudi and Iceberg tables.

- How it works: The index is derived metadata: build or drop it without rewriting the table’s data files. It groups vectors into clusters, stores compact quantized codes, and is maintained by the same commits that write the table.

- Why it matters: On 247 million vectors, warm queries return in 63.5 ms at the median. Lakegres is about 10 times faster than Databricks AI Search at the warm median.

Today we are announcing vector search in Onehouse Lakegres, with a native index stored inside existing…

Technology