Artificial Intelligence1 min read

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI

By · Published by Everything Blog

In short

The Agentforce Foundation team at Salesforce previously relied on an internal gateway using basic health checks and public networking paths, which lacked remote capacity signals and elevated failure risks. To resolve these challenges, the team integrated Amazon SageMaker AI inference routers to provide a single, managed invocation path across multi-Region endpoints. Applications target a logical model name mapped to specific Amazon Resource Names. The system dynamically redirects workloads away from local endpoints when timeouts, throttles, platform faults, or container errors signal emerging capacity constraints.

Key points

  • Managed inference routing eliminates capacity fragmentation by balancing workloads across…: Managed inference routing eliminates capacity fragmentation by balancing workloads across multiple AWS Regions and accounts.
  • Prioritizing local endpoints while shifting traffic during disruptions maintains service…: Prioritizing local endpoints while shifting traffic during disruptions maintains service reliability without violating data residency commitments.
  • Replacing custom routing logic with designated Amazon Resource Name targets reduces opera…: Replacing custom routing logic with designated Amazon Resource Name targets reduces operational overhead across diverse enterprise applications.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI

Thanks to Amit Gupta, Senior Director- AI/ML and Mark Wang, Software Engineering PMTS from Salesforce for your contributions in the vision and partnership of cross-Region inference routing on SageMaker AI.

Salesforce’s Agentforce Foundation team serves a wide variety of models for products across Salesforce, including Agentforce and the Salesforce Trust Layer, in 30+ global Regions. These include both models trained in house and open source models. Hosting models separately in each Region splits capacity and makes traffic spikes harder to absorb, which turns a capacity shortfall into an availability problem.

This post shows how Salesforce set up the cross-Region inference routing in SageMaker AI and how it lets the team add capacity securely while keeping its data residency commitments. It also walks through how Salesforce sees where each request was served and why. The cross-Region inference routing gives an application one invocation path across SageMaker AI endpoints in several Regions and accounts, and sends each request to an endpoint that can serve it.…

Artificial Intelligence