Artificial Intelligence1 min read

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

By · Published by Everything Blog

In short

The architecture allows engineering teams to separate prefill and decode tasks across dedicated pools of workers linked by high-performance networking, or maintain them in unified environments. AWS Deep Learning Containers supply the runtime, enabling compatibility with popular inference engines including vLLM, SGLang, and TensorRT-LLM. Dynamo's routing system tracks prefix caching and active hardware loads to protect nodes from being overloaded. Automated cluster management also handles node replacements and recreates failed workers to maintain service stability.

Key points

  • Deployments support disaggregated architectures that isolate prefill and decode workloads…: Deployments support disaggregated architectures that isolate prefill and decode workloads across separate hardware pools.
  • Teams can implement workloads using either custom DynamoGraphDeployment manifests or Open…: Teams can implement workloads using either custom DynamoGraphDeployment manifests or OpenAI-compatible InferenceEndpointConfig resources.
  • Automated health monitoring detects system failures to replace compromised nodes and recr…: Automated health monitoring detects system failures to replace compromised nodes and recreate affected workers without manual intervention.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge

instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.

How the integration works

The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and…

Artificial Intelligence