Technology1 min read

Detect, act, keep the evidence: node reliability across three clouds

By · Published by Everything Blog

In short

The effort began after subtle issues like undetected RAID array drive failures on AWS EC2 degraded systems and required manual intervention from human operators. Standard Kubernetes monitors routinely overlook specialized hardware faults, hung kernel tasks, and read-only filesystems. ClickHouse Cloud deployed a dedicated framework across all supported clouds to standardize monitoring, remediation, and evidence gathering. The resulting system surfaces hidden issues as Kubernetes conditions, automatically evicts workloads, and stores forensics before deleting instances.

Key points

  • Default cloud provider health checks frequently fail to catch broken disks and degraded h…: Default cloud provider health checks frequently fail to catch broken disks and degraded hardware states.
  • The Node Problem Mitigator automatically taints, drains, and terminates identified proble…: The Node Problem Mitigator automatically taints, drains, and terminates identified problematic instances across all clouds.
  • The Node Debug Collector captures essential forensics and system states before damaged no…: The Node Debug Collector captures essential forensics and system states before damaged nodes are permanently deleted.

Sources checked by our analysis pipeline: 1

ClickHouse Cloud runs on many thousands of Kubernetes nodes across AWS, GCP, and Azure. At that scale, a node with a broken disk, a stuck kernel task, or a half-working network is a daily event. Most of these failures do not look like a node failure at first. Kubernetes still reports the node as Ready. The cloud provider console shows the instance as running. What you see instead is a ClickHouse replica that cannot write to its cache disk, a pod that stays in ContainerCreating, or a query that times out on one replica and works on the others.

While cloud service providers (CSPs) sometimes try to detect and fix these issues, they are not succeeding at these in 100% of cases. Also, the “fix” behavior differs between cloud providers and is not always the optimal solution for a given issue. We wanted to have more control and consistency across CSPs. Hence, we started a project we called Compute Layer Reliability.

This post explains what we found when we looked at how node health was handled on each cloud, why…

Technology