Technology1 min read

How we raised Databricks cluster rate limits by 7.5×

By · Published by Everything Blog

In short

Databricks updated its classic compute platform across Amazon Web Services, Microsoft Azure, and Google Cloud by adopting adaptive concurrency controls. Long-running cluster creation processes previously caused unfinished jobs to accumulate rapidly whenever underlying cloud providers experienced delays. By measuring in-flight work and system health, the platform automatically scales task intake up or down depending on active conditions. A fixed API request-rate ceiling remains in place as a final safeguard against extreme traffic surges.

Key points

  • Adaptive concurrency limits dynamically regulate intake based on downstream system perfor…: Adaptive concurrency limits dynamically regulate intake based on downstream system performance and the volume of unfinished work.
  • Automated throttling helps prevent cascading retry storms that occur when cloud provider…: Automated throttling helps prevent cascading retry storms that occur when cloud provider slowdowns trigger request timeouts.
  • Databricks continues to enforce a fixed request ceiling as a backup guardrail against abn…: Databricks continues to enforce a fixed request ceiling as a backup guardrail against abnormal traffic spikes.

Sources checked by our analysis pipeline: 1

Why adaptive concurrency limits work better than fixed request-rate limits for long-running operations.

by Heqing Ya, Tan Zhang, Ankit Batra and Abhishek Rai

As part of the Compute Platform team at Databricks, we turn cloud infrastructure into clusters ready to run customer workloads. Our day-to-day work includes provisioning virtual machines and coordinating cluster creation across AWS, Azure, and Google Cloud. Our goal is to keep these systems available while letting customers use as much capacity as we can safely provide.

In this post, we'll explore why fixed request-rate limits can fall short for these long-running operations, how adaptive concurrency limits help, and what we learned applying them to cluster creation for Databricks classic compute. You don't need to know any of this to use Databricks, but if you're curious about what happens behind a Databricks Runtime cluster launch, join us for a look inside the kitchen.

Historically, we protected the system with fixed request-rate limits, tested against different load patterns and failure scenarios. We scaled our services primarily by adding resources and adjusting per-customer limits.

The combination of workloads and…

Technology