Artificial Intelligence1 min read

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

By · Published by Everything Blog

In short

The enhancements to Amazon SageMaker HyperPod introduce three distinct capabilities targeting operational failures across distributed artificial intelligence workloads. MultiNodeTrainingValidation runs test models on Amazon Elastic Kubernetes Service clusters to uncover invalid loss values and stalls during setup or on demand. Simultaneously, a dedicated monitoring agent tracks Kubernetes issues such as tainted nodes and missing DaemonSets, sending alerts directly to CloudWatch logs. For Slurm-based environments, an on-node agent can cancel unresponsive jobs to enable automatic requeuing and restore cluster efficiency.

Key points

  • Amazon SageMaker HyperPod now detects cluster-level disruptions and silent failures that…: Amazon SageMaker HyperPod now detects cluster-level disruptions and silent failures that standard individual node checks fail to catch.
  • Automated detection enables Slurm to cancel and automatically requeue unresponsive jobs t…: Automated detection enables Slurm to cancel and automatically requeue unresponsive jobs to optimize overall computing resources.
  • Deep health checks and continuous resource monitoring report critical pod and node issues…: Deep health checks and continuous resource monitoring report critical pod and node issues directly to Amazon CloudWatch logs.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.

SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look…

Artificial Intelligence