Technology1 min read

Adaptive Integer Compression in TimescaleDB: Built for the Read Path

By · Published by Everything Blog

In short

The new system selects from six fast-decompressing encodings for blocks of up to 256 values, relying on a tailored version of FastLanes for bit-packing. In benchmark tests, isolated integer decompression speeds jumped by five to seven times, which translated into approximately 50% faster query execution overall. Although integer compression ratios rose by an average of 50%, total database storage decreased by roughly 10% after factoring in elements like TOAST tables, non-integer columns, index overhead, and tuple-level costs. DeltaDelta previously relied on Simple8bRLE, which lacked the ability to leverage modern CPU SIMD vectorization capabilities.

Key points

  • The previous DeltaDelta system relied on Simple8bRLE and sequential dependencies that pre…: The previous DeltaDelta system relied on Simple8bRLE and sequential dependencies that prevented modern CPU SIMD vectorization.
  • The new approach dynamically selects one of six encodings per 256-value block using a tai…: The new approach dynamically selects one of six encodings per 256-value block using a tailored FastLanes bit-packing implementation.
  • Non-time-series distributions such as foreign keys, flags, status codes, and interleaved…: Non-time-series distributions such as foreign keys, flags, status codes, and interleaved data are handled much more effectively by the new design.

Sources checked by our analysis pipeline: 1

AIC replaces DeltaDelta for integer columns in TimescaleDB. It picks one of six encodings for each block of up to 256 values, using only encodings that decompress fast. In our tests, databases come out about 10% smaller and typical queries run about 50% faster. It is off by default in 2.31.0 and planned to be on by default in 2.32.0.

DeltaDelta (DD) is TimescaleDB's current integer compression. It stores each value as the difference between consecutive differences, then packs those small numbers with the Simple8bRLE algorithm. It works very well on steadily moving values. On a lot of the integer data in a real database, it doesn't, and its decompression is slower than it needs to be.

Adaptive Integer Compression (AIC) is its replacement. In version 2.31.0, TimescaleDB released AIC as a preview, and 2.32.0 is planned to enable it by default.

Storage: AIC compresses data about 10% better than DD, measured at the database level on real-life, benchmark and synthetic datasets.

Queries: typical database queries get around 50% faster.

Decompression: integer decompression is 5-7x faster, measured in isolation.

Integer…

Technology