Previously, Amazon SageMaker users had to provision an entire physical accelerator for an endpoint, which tied model counts directly to total hardware volume and incurred costs for unused capacity. The new partition capability enables one GPU to be split into as many as seven separate instances, each allocated its own compute, cache, memory, and bandwidth. Because models remain continuously loaded in their dedicated hardware slices, endpoints avoid cold-start delays while maintaining consistent latency. However, these configuration layouts must be defined during endpoint creation and can only be modified when partitions are idle.
Key points
Hardware-level partitioning eliminates cold-start latency penalties by keeping distinct m…: Hardware-level partitioning eliminates cold-start latency penalties by keeping distinct models permanently resident within their assigned slices.
Errors or unexpected traffic spikes affecting one partition are strictly contained and wi…: Errors or unexpected traffic spikes affecting one partition are strictly contained and will not disrupt adjacent workloads on the physical GPU.
Teams hosting small to mid-sized models can size infrastructure more accurately instead o…: Teams hosting small to mid-sized models can size infrastructure more accurately instead of paying for unneeded full accelerator capacity.
Sources checked by our analysis pipeline: 1
Artificial Intelligence
SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator.
This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge
has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach.
Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced…
State-level voter restrictions on energy- and water-intensive data centres have triggered sharp sell-offs in semiconductor and artificial intelligence infrastructure shares. These sudden regulatory hurdles have reignited broader market fears over circular financing, questionable revenues, and unsustainable debt structures across the sector. Heavily leveraged firms now face prolonged development timelines and surging capital costs to secure alternate power and operational sites.
OpenAI has secured $11 billion in new funding, according to a report from Bloomberg News. The investment values the artificial intelligence company at $1.25 trillion.
The artificial intelligence company has completed the initial portion of a massive investment round reported on Friday. This injection of fresh capital marks a major financial milestone as the firm continues advancing its broader financing efforts.