Standard metrics masked severe performance issues because typical monitoring tools only record the share of time a kernel is active rather than full workload dynamics. Developers discovered that llama.cpp was transferring 643 megabytes of state checkpoints from the GPU to the host on every single request. Disabling these transfers nearly doubled processing throughput from 58 to 110 tokens per second while boosting GPU activity to 88 percent. Subsequent updates to newer llama.cpp builds further mitigated these transfer penalties by optimizing how state data is managed.
Key points
Disabling state transfers nearly doubles inference speeds but eliminates the performance…: Disabling state transfers nearly doubles inference speeds but eliminates the performance benefits associated with prompt reuse during returning conversations.
Low reported GPU utilization can indicate serialization and memory transfer bottlenecks r…: Low reported GPU utilization can indicate serialization and memory transfer bottlenecks rather than an underutilized hardware capacity.
Upstream software updates to llama.cpp successfully reduced data transfer penalties by 67…: Upstream software updates to llama.cpp successfully reduced data transfer penalties by 67 to 75 percent under default system configurations.
Sources checked by our analysis pipeline: 1
Your GPU says 35% busy. Busy doing what?
nvidia-smi said our H100 was 35% busy while users queued for it. Counting every kernel and memory copy found the cause, 643 MB copied off the GPU per request.
- H100 and L4 · llama.cpp and vLLM
- llama.cpp
- vLLM
- GPU utilisation
- CUPTI
We put an LLM server on an H100, pointed sixteen clients at it, and opened nvidia-smi
. It said 35% busy. Throughput was 58 tokens per second, and users waited almost two seconds for a first token.
So the card sat idle two thirds of the time while people queued for it. None of the usual tools could say where the rest of the time went.
This is how we found out, and the small tool we built to do it.
The percentage says less than you think
“GPU busy” sounds like “how hard the GPU is working”. It is not. NVIDIA documents it as the share of each sampling period in which at least one kernel was running. A kernel is one small program the…
Since launching in November 2006, the broadcaster has focused on underreported regions and human-centered reporting amid major global conflicts and political crises. As the international media landscape grapples with the rise of artificial intelligence and misinformation, the network is introducing an updated on-air presentation to anchor its future coverage.
The artificial intelligence company is pursuing public market capital to satisfy massive infrastructure demands, including data centers, electricity, and specialized processors. Operating as a public benefit corporation valued well above $1 trillion, the firm aims to deploy systems capable of solving major global problems. However, leadership balances this ambitious vision against catastrophic hazards that could destabilize global security and modern economic systems.
Modern automated programming tools rely on models trained by scraping decades of freely shared software from platforms like Stack Overflow and the GNU Project without proper attribution. Converting these outputs into closed, proprietary products essentially functions as license laundering while contradicting United States legal precedents on human authorship. Establishing standard open-access practices protects the collaborative software ecosystem that made automated coding possible in the first place.