Technology1 min read

PostgreSQL + ClickHouse Recovery: Test the Whole Data Service

By · Published by Everything Blog

In short

Relying strictly on basic uptime metrics or row counts can leave business dashboards displaying inaccurate figures even after databases come back online. To ensure full operational readiness, verification procedures must evaluate availability, recovery points, elapsed recovery time, data freshness, and correctness. Correctness should be checked against an independent expected-results file rather than deriving verification data from the databases themselves. Furthermore, ClickHouse must actively consume streaming updates until consumer lag drops to zero against the authoritative PostgreSQL source.

Key points

  • Restoring operational databases without validating the full streaming pipeline risks serv…: Restoring operational databases without validating the full streaming pipeline risks serving corrupted or lagging data to reporting dashboards.
  • Correctness testing must compare queries against external expected-results files rather t…: Correctness testing must compare queries against external expected-results files rather than trusting figures generated directly by the databases.
  • Application owners rather than infrastructure administrators are ultimately responsible f…: Application owners rather than infrastructure administrators are ultimately responsible for confirming that recovered query results match business requirements.

Sources checked by our analysis pipeline: 1

blog

PostgreSQL + ClickHouse Recovery: Test the Whole Data Service

A restore job finishes green. PostgreSQL is taking orders, ClickHouse is answering queries, and the revenue dashboard loads without errors. Then someone from finance asks why January 15 shows 12,750 in revenue when the order system says 12,350.

Nothing is down and no alert has fired. The numbers are simply wrong.

This post walks through a recovery exercise for one specific architecture. PostgreSQL is the system of record for a fictional order service. Debezium captures its changes (change data capture, or CDC) and streams them through Apache Kafka, and ClickHouse serves the revenue dashboards. In the scenario, the ClickHouse tier is lost while PostgreSQL keeps accepting writes, and the question is what evidence you need before telling the business that the analytics service is back.

Not every PostgreSQL + ClickHouse deployment fails like this, and this post doesn’t try to cover every pattern. For the broader catalog, read PostgreSQL + ClickHouse architectural patterns. This post is narrower: it builds a recovery test that fails when the data is wrong.

Define…

Technology