Technology1 min read

It's the Memory Latency, Stupid

By · Published by Everything Blog

In short

Modern processors can execute several instructions per nanosecond, yet retrieving data directly from system RAM takes roughly 50 to 100 nanoseconds. Although fast integrated caches mitigate this delay, extensive database operations like broad aggregations cause frequent cache misses that force slow memory lookups. In benchmark testing on an AMD EPYC server, running a basic single-core aggregation query across millions of distinct entries took nearly four seconds primarily due to these memory transfer delays.

Key points

  • Reported 100% CPU utilization does not guarantee full core productivity because processor…: Reported 100% CPU utilization does not guarantee full core productivity because processors frequently stall while waiting for RAM.
  • High-volume database operations regularly exceed the storage capacity of integrated hardw…: High-volume database operations regularly exceed the storage capacity of integrated hardware caches, triggering performance-degrading cache misses.
  • System processing speeds during large-scale aggregations are fundamentally bounded by mem…: System processing speeds during large-scale aggregations are fundamentally bounded by memory latency rather than raw instruction execution limits.

Sources checked by our analysis pipeline: 1

Making GROUP BY faster by doing more (CPU!) work

When you see your databases CPU at 100%, what is it actually doing? We like to act as though CPUs run code linearly for simplicity's sake, but in truth CPUs are incredibly complex beasts; they can potentially do many instructions at once, or sometimes none at all (despite our console showing 100%). Because CPUs can do so many things, one CPU at 100% may be doing much less than another at 100%. A CPU is considered "idle" only if no task (Linux task) needs CPU time; this does not mean the CPU itself is being fully saturated. So why would a CPU not be fully utilized, even if it appears to be at 100%?

The answer is usually memory.

Memory accesses are orders of magnitude slower than internal CPU operations. To give a good picture of how much so, a modern CPU might be able to process somewhere in the realm of 4-8 instructions per nanosecond. A memory access to RAM can take somewhere in the realm of 50-100ns. To ease…

Technology