Tiered key-value cache offloading in Red Hat OpenShift AI 3.5 preserves model context outside accelerator memory to enhance long-context performance on NVIDIA H100 systems. Evaluated with Nemotron-Ultra-253B on workloads averaging approximately 50,000 input tokens, the architecture reduced the mean time to first token from 60 seconds down to 10 seconds while improving overall throughput by up to 5.8x. When working sets expand beyond host CPU memory, incorporating a secondary layer like NVMe or CephFS nearly doubles the throughput gains achieved by CPU offloading alone.
Key points
Secondary offloading to NVMe or CephFS nearly doubles throughput gains once memory demand…: Secondary offloading to NVMe or CephFS nearly doubles throughput gains once memory demands exceed available host CPU capacity.
Local NVMe delivers the lowest secondary latency on single nodes, whereas CephFS facilita…: Local NVMe delivers the lowest secondary latency on single nodes, whereas CephFS facilitates cross-replica reuse across multi-node clusters.
System operators must pinpoint the operational threshold where cache retrieval latency pr…: System operators must pinpoint the operational threshold where cache retrieval latency provides faster execution than prompt recomputation.
Sources checked by our analysis pipeline: 1
Long-context workloads are quickly becoming the norm for enterprise AI. Agentic applications carry an ever-growing history of tool calls and intermediate results, retrieval-augmented generation prepends thousands of tokens of retrieved context, and multi-turn assistants resend the whole conversation with every turn. In all of these cases, the model sees large parts of the same prompt again and again.
That repetition should be cheap, because the model has already done the work once. In practice, it often isn't. The state that makes reuse possible, the KV cache, lives in GPU memory, and GPU memory runs out long before the context does. When it runs out, the work is discarded and recomputed from scratch the next time the same prefix shows up.
In Red Hat OpenShift AI 3.5, the inference platform offloads that state from GPU memory into cheaper, higher-capacity tiers: CPU DRAM, local NVMe, or any POSIX-compliant distributed and parallel file system. In this blog post, we walk through how we evaluated those tiers (using CephFS as our shared file system throughout), starting from raw storage latency, translating it into a…
The unexpected restriction follows surging operational expenditures tied to running the sophisticated system. Leadership intervened after determining that maintaining previous access tiers was financially unfeasible under current resource constraints. The move directly limits how frequently users can deploy the advanced tool for automated tasks.
The artificial intelligence developer has submitted new regulatory disclosures ahead of a potential public debut targeted for late this month. These updated financial records offer an unvarnished look at the immense compute expenses that completely offset the firm's explosive top-line growth. Investors are now assessing whether this intensive capital spending will impede the company's path to sustained profitability.
The artificial intelligence developer achieved a significant financial milestone by generating substantial quarterly business. This performance coincides with the organization's preparation for a landmark technology debut. The results underscore accelerating commercial momentum as the company approaches its next major product release.