Artificial Intelligence1 min read

Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

By · Published by Everything Blog

In short

The observability framework supports workloads operating on both Amazon Elastic Kubernetes Service and Slurm cluster orchestrators. Telemetry collection is divided into a low-overhead default tier gathered every 30 seconds alongside an opt-in advanced tier for deeper network and host statistics. All metrics flow directly into Amazon CloudWatch or Amazon Managed Service for Prometheus, enabling automated outlier analysis alongside static rule evaluations. This dual approach immediately surfaces known hardware errors and anomalous behavior across individual accelerators.

Key points

  • Training workloads orchestrated through Amazon EKS or Slurm benefit from reduced compute…: Training workloads orchestrated through Amazon EKS or Slurm benefit from reduced compute loss by rapid isolation of degraded infrastructure.
  • Advanced telemetry options allow operators to track detailed host behavior, complete acce…: Advanced telemetry options allow operators to track detailed host behavior, complete accelerator metrics, and Elastic Fabric Adapter network statistics.
  • Predefined diagnostic rules instantly identify critical hardware faults including thermal…: Predefined diagnostic rules instantly identify critical hardware faults including thermal throttling, degraded throughput, and ECC memory or XID failures.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

Large-scale model training is one of the most expensive workloads you can run on GPUs, and a single unhealthy accelerator can stall an entire distributed job. When training throughput drops, the hard part is rarely noticing that something is wrong. It is figuring out why and where. Teams often spend hours or days sifting through logs and disconnected dashboards to decide whether a slowdown came from a throttled GPU, a saturated network link, or a stalled data pipeline on the host.

Today, we are announcing enhanced observability in Amazon SageMaker HyperPod, a new capability that helps you diagnose GPU training issues faster. It brings together diagnostic and profiling capabilities for training workloads that run on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators:

Diagnostic summary uses Amazon CloudWatch’s DetectGroupOutliers API to compare each GPU in a training job on utilization, memory, temperature, power, and SM clock, and flags GPUs that deviate from their peers. Static diagnostic rules surface common failure patterns such as GPU hardware faults (XID errors,…

Artificial Intelligence