Technology1 min read

Smarter cost controls for serverless

By · Published by Everything Blog

In short

Databricks is expanding financial and operational governance for serverless compute across notebooks, jobs, and pipelines to help teams manage cloud budgets. Spend attribution is handled by beta tags that send data to billing tables for analysis through SQL queries or console dashboards. Access controls and budget alerts are generally available across AWS, Azure, and Google Cloud. By default, workspaces grant all users "Can Use" permissions until administrators modify them to restrict access.

Key points

  • Administrators can analyze serverless expenditures by user, product, and tag using SQL qu…: Administrators can analyze serverless expenditures by user, product, and tag using SQL queries or account console dashboards.
  • Serverless Compute Access Controls allow platform teams to assign "Can Use" and "Can Mana…: Serverless Compute Access Controls allow platform teams to assign "Can Use" and "Can Manage" permission levels.
  • All workspace users receive "Can Use" serverless permissions by default until administrat…: All workspace users receive "Can Use" serverless permissions by default until administrators restrict execution to approved groups.

Sources checked by our analysis pipeline: 1

Bringing familiar cost controls to serverless compute

by Piyush Singh and Vijay Raja

At Databricks, our serverless compute usage has grown more than 3x in the past year, and thousands of customers now run mission critical notebooks, jobs, and pipelines on it. As adoption climbs, the question every platform team asks first only gets louder: how do we gain better visibility into serverless costs and keep the cost within our budget?

Databricks customers running serverless SQL and Model Serving already attribute spend, govern access, and monitor consumption without touching a single cluster knob. We are now bringing that same functionality to serverless compute for notebooks, jobs, and pipelines. We are making serverless cost controls work the way platform teams already expect it to.

On classic compute, cost control lived in the infrastructure. You capped spend by capping machines: node counts, instance types, autoscaling ranges, and cluster policies that fenced in what users could provision. The controls were required because you had to manage the infrastructure on your own.

Serverless takes those knobs away, on purpose. Provisioning, scaling, and right-sizing infrastructure…

Technology