The mechanism works by having the model container report cache states to a managed cache index, which guides the endpoint's routing decisions. The managed index utilizes one-way truncated hashes to securely track growing multi-turn conversations and record when entries are stored or evicted. When routing incoming traffic, the endpoint weighs the longest cached prompt match against current instance workloads. If the primary target instance is overloaded, the system smoothly redirects the query to the next-best candidate or reverts to standard load-aware routing.
Key points
Real-time endpoints reduce repetitive GPU processing by directing queries to instances wi…: Real-time endpoints reduce repetitive GPU processing by directing queries to instances with matching cached memory.
Inference workloads automatically balance between the length of the cached match and the…: Inference workloads automatically balance between the length of the cached match and the availability of instance resources.
Client-facing APIs do not need any modifications to take advantage of the new KV cache-aw…: Client-facing APIs do not need any modifications to take advantage of the new KV cache-aware routing mechanism.
Sources checked by our analysis pipeline: 1
Artificial Intelligence
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy
to KV_AWARE
in the endpoint configuration. Your clients call the endpoint with the same APIs as before.
Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In…
The tech giant's latest strategic shift repositions its enterprise software suite directly into an integrated artificial intelligence environment. By embedding essential productivity applications into this single hub, the company aims to fundamentally change how employees navigate daily digital tasks. Future updates will also incorporate persistent automation to handle tasks continuously across the platform.
Relying on large language models for routine operations like input validation, routing, and formatting introduces hundreds of milliseconds of latency and output variability. Engineering organizations are curbing these production expenses by adopting a structured evaluation framework that reserves expensive calls exclusively for ambiguous generation tasks. Shifting rule-based logic to deterministic code restores auditable, stable behavior while driving response times down to single-digit milliseconds.
Airline operations platform Flightscape encountered context window exhaustion and 15-second latency spikes when running a 40-tool scheduling agent on massive data payloads. To fix these operational bottlenecks, engineers designed an offloading framework that moves heavy structured JSON out of direct conversation turns and into external storage. This architectural pivot significantly decreases computational overhead while sustaining multi-turn reasoning on complex enterprise workloads.