Technology1 min read

MongoDB CDC connector for ClickPipes is now generally available

By · Published by Everything Blog

In short

The integration allows teams to continuously sync operational data into ClickHouse Cloud using MongoDB Change Streams. To optimize performance, the system divides collections into balanced ID ranges to execute multi-threaded initial loads and fast resynchronizations. Users can configure the pipeline programmatically through Terraform or the Cloud API, and choose structured ingestion to automatically map document fields into strongly typed columns. Unmapped or unexpected properties are routed into a dedicated JSON column to prevent data loss without backfills.

Key points

  • Operational databases can offload heavy analytical query traffic to ClickHouse Cloud thro…: Operational databases can offload heavy analytical query traffic to ClickHouse Cloud through continuous Change Streams replication.
  • Programmatic management is fully supported via Terraform integration and the ClickHouse C…: Programmatic management is fully supported via Terraform integration and the ClickHouse Cloud API.
  • Unrecognized schema fields are preserved within a dedicated JSON column to eliminate the…: Unrecognized schema fields are preserved within a dedicated JSON column to eliminate the need for data backfills.

Sources checked by our analysis pipeline: 1

Summary

Replicate data from MongoDB into ClickHouse Cloud in just a few clicks for blazing-fast analytics on document-based data. Now with parallel snapshotting, structured ingestion mode, and programmatic configuration (Cloud API, Terraform).

When we set out to develop a CDC connector for MongoDB, it felt like an instant home run: the semi-structured nature of documents was a perfect fit for ClickHouse’s native JSON data type, ClickHouse’s columnar architecture delivered up to 100x faster analytical queries, and continuous replication with ClickPipes meant you could now get insights from your operational MongoDB data in seconds, not hours.

But a new CDC connector is never an instant home run. Someone will reshard their MongoDB cluster during business hours. Someone will bulk-delete a few billion documents to enforce a retention policy. Someone will swear a field can never be null. After moving over 1.5PB of data for hundreds of production customers and hitting all sorts of edge cases, we’re ready to announce that the MongoDB CDC connector is now Generally Available!

We replaced our Fivetran-based batch pipeline with MongoDB ClickPipes, streaming 250 collections across…

Technology