Technology1 min read

Using NVIDIA Run:ai GPU Fractioning on Amazon EKS

By · Published by Everything Blog

In short

The platform relies on the modular KAI Scheduler and GPU Fractioning Operator to manage workload orchestration and hardware metrics on Amazon Elastic Kubernetes Service. During testing on an Amazon EC2 g5.xlarge instance running an NVIDIA A10G GPU, system nodes isolated workloads through defined memory contracts. The deployment permitted concurrent 6 GiB allocations under an 8 GiB limit while intercepting and blocking an out-of-bounds 9 GiB request. While this architecture enhances container density, it does not guarantee compute reservations or lower operational expenses.

Key points

  • The system enables concurrent workloads like interactive notebooks and model evaluations…: The system enables concurrent workloads like interactive notebooks and model evaluations on GPUs lacking hardware-level partitioning.
  • Software-enforced memory limits provide distinct error domains but do not establish stric…: Software-enforced memory limits provide distinct error domains but do not establish strict security boundaries or proportional compute reservations.
  • Increasing container density through this method does not inherently assure universal cos…: Increasing container density through this method does not inherently assure universal cost reductions or improved application throughput.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Using NVIDIA Run:ai GPU Fractioning on Amazon EKS

GPU sharing is not new: Multi-Instance GPU (MIG), time-slicing, and Multi-Process Service (MPS) address different isolation and utilization needs. NVIDIA Run:ai GPU Fractioning builds on MPS with Kubernetes-integrated scheduling, per-Pod memory contracts, enforcement, and metrics.

NVIDIA Run:ai is AI workload orchestration software for Kubernetes. Its GPU Fractioning Operator can help several teams share one or two GPUs on Amazon Elastic Kubernetes Service (Amazon EKS). Notebooks, model evaluations, and small inference services can leave teams waiting even when GPU memory remains available. In this walkthrough’s baseline, the NVIDIA device plugin exposes the single GPU as one nvidia.com/gpu

resource, with GPU sharing disabled. Two Pods each requesting nvidia.com/gpu: 1

cannot run concurrently on that node, even when their combined GPU-memory demand fits.

NVIDIA Run:ai GPU Fractioning adds Kubernetes scheduling integration, enforced GPU-memory limits, and per-Pod metrics to software-based sharing. The approach applies to NVIDIA GPUs that meet the GPU Fractioning Operator’s hardware and software requirements, including GPUs without MIG. We use one Amazon Elastic Compute Cloud (Amazon EC2) g5.xlarge instance with an…

Technology