Artificial Intelligence1 min read

SuperEMeans: Making e-means 3× faster

By · Published by Everything Blog

In short

Developers combined the dimension-skipping approach of SuperKMeans with the rapid convergence of e-means to accelerate vector clustering. The hybrid method matches Lloyd’s k-means quality in only two epochs rather than the conventional ten. In practical evaluations, the software ran efficiently on multi-core systems without degrading vector recall metrics. Cross-platform support ensures broad usability on diverse CPU architectures such as Intel, Zen, and Graviton.

Key points

  • The combined method achieves up to a 3× overall acceleration and runs 1.8× faster than st…: The combined method achieves up to a 3× overall acceleration and runs 1.8× faster than standalone e-means.
  • Pruning unnecessary vector dimensions allows systems to reduce compute overhead while pre…: Pruning unnecessary vector dimensions allows systems to reduce compute overhead while preserving standard recall@100 accuracy.
  • Broad CPU optimization enables high-performance vector clustering across Intel, Graviton,…: Broad CPU optimization enables high-performance vector clustering across Intel, Graviton, and Zen hardware environments.

Sources checked by our analysis pipeline: 1

TL;DR: We fused SuperKMeans and e-means, resulting in 3× faster clustering of high-dimensional vectors without quality trade-offs. Available now at https://github.com/cwida/SuperKMeans.

Recently, there has been a lot of buzz around clustering of vector embeddings: EVōC, SuperKMeans, QGKmeans, and just last week, e-means, o-means, and s-means. Among these, e-means is a big deal: it promises the same quality as Lloyd’s with only 2 epochs. Why is this a big deal for vector search?

Well, clustering is a key building block for large-scale vector indexes (e.g., IVF, ScANN), which is precisely the bottleneck that users have to wait on before issuing similarity search queries.

A year ago, we released SuperKMeans. SuperKMeans accelerates Lloyd’s k-means by 2—10× by avoiding scanning vector dimensions while clustering, without trade-offs in clustering quality. SuperKMeans has been rapidly adopted by industry (ParadeDB, Zilliz, FAISS), showing how timely it is to improve clustering performance for high-dimensional vectors. However, e-means is also exciting because it tackles effort by running fewer epochs. Traditionally, clustering for vector embeddings is standardized to 10 epochs. e-means promises the same (or better) clustering quality with…

Artificial Intelligence