Artificial Intelligence1 min read

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

By · Published by Everything Blog

In short

The mechanism works by having the model container report cache states to a managed cache index, which guides the endpoint's routing decisions. The managed index utilizes one-way truncated hashes to securely track growing multi-turn conversations and record when entries are stored or evicted. When routing incoming traffic, the endpoint weighs the longest cached prompt match against current instance workloads. If the primary target instance is overloaded, the system smoothly redirects the query to the next-best candidate or reverts to standard load-aware routing.

Key points

  • Real-time endpoints reduce repetitive GPU processing by directing queries to instances wi…: Real-time endpoints reduce repetitive GPU processing by directing queries to instances with matching cached memory.
  • Inference workloads automatically balance between the length of the cached match and the…: Inference workloads automatically balance between the length of the cached match and the availability of instance resources.
  • Client-facing APIs do not need any modifications to take advantage of the new KV cache-aw…: Client-facing APIs do not need any modifications to take advantage of the new KV cache-aware routing mechanism.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy

to KV_AWARE

in the endpoint configuration. Your clients call the endpoint with the same APIs as before.

Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In…

Artificial Intelligence