Technology1 min read

HubSpot Incident Report: September 8, 2026

By · Published by Everything Blog

In short

The incident began when an automated safeguard terminated an oversized permissions update job, allowing the cache to exceed its 10-minute staleness limit. This triggered direct database queries for authorization checks that quickly overwhelmed service thread pools. In addition to temporarily raising the staleness threshold to recover systems, HubSpot implemented permanent fixes including query pagination and the removal of direct database fallbacks in favor of a dedicated read cache.

Key points

  • Direct database fallbacks for authorization checks caused thread pool exhaustion once the…: Direct database fallbacks for authorization checks caused thread pool exhaustion once the permissions cache exceeded its staleness limit.
  • Technical teams recovered the impacted platforms by manually increasing the cache stalene…: Technical teams recovered the impacted platforms by manually increasing the cache staleness threshold while restarting the background update job.
  • Future safeguards include paginating background jobs, deploying a dedicated read cache, a…: Future safeguards include paginating background jobs, deploying a dedicated read cache, auditing critical services, and improving staleness alerting.

Sources checked by our analysis pipeline: 1

On September 8, 2026, between 11:40 AM and 12:19 PM EDT (UTC-4), HubSpot customers in the North America region experienced an outage across HubSpot tooling. No data was lost during this incident. After completing a thorough analysis of this incident, here is a detailed account of what happened, why it occurred, and the steps we are taking to prevent similar issues in the future.

What Happened

On the morning of Tuesday, September 8th, a job that updates our permissions cache, which is used to determine user access to specific HubSpot features, began to fail. This resulted in our permissions cache exceeding a staleness threshold that caused the permissions service to fall back to querying the underlying database for every authorization check.

Because these database queries are substantially slower than cache lookups, the permissions service had to process more requests in parallel, eventually exhausting thread pools on the service.

Our internal reliability automation alerted the engineering team to the elevated error rates within a minute of the impact starting. An internal incident was declared 4 minutes thereafter. However, the root cause…

Technology