Artificial Intelligence1 min read

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

By · Published by Everything Blog

In short

Exa initiated these infrastructure improvements when it prepared to re-embed 30 billion high-priority documents using a new model architecture. Running two simultaneous pipelines required the company to deploy single-core workloads inside Kubernetes pods across Inferentia2 and Trainium2 chips to maintain container consistency. Technical refinements included memory profiling with Neuron Explorer to resolve spill and reload bottlenecks between high-bandwidth memory and the Tensor Engine. Together with weight layout adjustments, these steps significantly improved multi-tower semantic model performance.

Key points

  • Re-embedding 30 billion documents temporarily doubled operational loads, making cost redu…: Re-embedding 30 billion documents temporarily doubled operational loads, making cost reduction per document a critical objective.
  • Compiler autocast allowed the infrastructure to operate at lower precision to accelerate…: Compiler autocast allowed the infrastructure to operate at lower precision to accelerate execution while fully preserving model accuracy.
  • Detailed memory profiling eliminated transfer bottlenecks between the Tensor Engine and h…: Detailed memory profiling eliminated transfer bottlenecks between the Tensor Engine and high-bandwidth memory to maximize hardware utilization.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

At Exa, we are building one of the few true web indexes by aggregating and serving knowledge from across the web, along with private and synthetic sources. We optimize the index for semantic search, grounding agents and humans in true, high-quality data. We use multi-tower semantic machine learning (ML) models to embed web data and query data into the same vector space, so users can express complex queries in natural language alongside traditional keyword search. The keystone technology behind this is an in-house vector database. It must ingest as much high-quality web data as possible and embed all of it. The unit economics of embedding therefore underlie everything we do. At the scale of tens or hundreds of billions of documents, finding the hardware that delivers the best performance per dollar isn’t a tuning detail. Any per-document cost improvement compounds into a significant number.

This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware…

Artificial Intelligence