Artificial Intelligence1 min read

Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI

By · Published by Everything Blog

In short

The inference router functions as a Regional resource that distributes application requests evenly among healthy local targets before failing over elsewhere. When a local environment degrades, incoming requests are automatically diverted to other designated endpoints and are returned gradually as recovery occurs. Each real-time response details which destination processed the request and the reason it was chosen. Organizations must continue independently provisioning models, autoscaling rules, and infrastructure, as the router does not deploy artifacts or create capacity.

Key points

  • Teams no longer need to develop custom routing infrastructure to maintain model serving d…: Teams no longer need to develop custom routing infrastructure to maintain model serving during regional GPU constraints.
  • Users must continue manually managing endpoint deployments, instance pools, and autoscali…: Users must continue manually managing endpoint deployments, instance pools, and autoscaling across their designated target regions.
  • Direct endpoint invocation remains functional alongside the router, and configured routin…: Direct endpoint invocation remains functional alongside the router, and configured routing targets fail over on an independent basis.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI

Serving a model from more than one AWS Region is a well-established pattern for highly available inference: when one Region runs short of GPU capacity, requests move to another. The pattern is simple to describe and hard to operate. Until now, if you served models on Amazon SageMaker AI managed endpoints, cross-Region routing was a piece you still had to build yourself.

Today, we’re introducing cross-Region inference routing in Amazon SageMaker AI. It uses a new resource, the inference router, that gives your application one invocation path across the endpoints and inference components you already run, in one or more AWS Regions and AWS accounts. You register the destinations that serve a model, and your application invokes the router instead of a specific endpoint. The router keeps requests in its own Region while the local destination is healthy, shifts new requests to another registered destination when it isn’t, and returns traffic gradually after recovery.

In this post, we explain how cross-Region inference routing…

Artificial Intelligence