The incident began when an automated safeguard terminated an oversized permissions update job, allowing the cache to exceed its 10-minute staleness limit. This triggered direct database queries for authorization checks that quickly overwhelmed service thread pools. In addition to temporarily raising the staleness threshold to recover systems, HubSpot implemented permanent fixes including query pagination and the removal of direct database fallbacks in favor of a dedicated read cache.
Key points
Direct database fallbacks for authorization checks caused thread pool exhaustion once the…: Direct database fallbacks for authorization checks caused thread pool exhaustion once the permissions cache exceeded its staleness limit.
Technical teams recovered the impacted platforms by manually increasing the cache stalene…: Technical teams recovered the impacted platforms by manually increasing the cache staleness threshold while restarting the background update job.
Future safeguards include paginating background jobs, deploying a dedicated read cache, a…: Future safeguards include paginating background jobs, deploying a dedicated read cache, auditing critical services, and improving staleness alerting.
Sources checked by our analysis pipeline: 1
On September 8, 2026, between 11:40 AM and 12:19 PM EDT (UTC-4), HubSpot customers in the North America region experienced an outage across HubSpot tooling. No data was lost during this incident. After completing a thorough analysis of this incident, here is a detailed account of what happened, why it occurred, and the steps we are taking to prevent similar issues in the future.
What Happened
On the morning of Tuesday, September 8th, a job that updates our permissions cache, which is used to determine user access to specific HubSpot features, began to fail. This resulted in our permissions cache exceeding a staleness threshold that caused the permissions service to fall back to querying the underlying database for every authorization check.
Because these database queries are substantially slower than cache lookups, the permissions service had to process more requests in parallel, eventually exhausting thread pools on the service.
Our internal reliability automation alerted the engineering team to the elevated error rates within a minute of the impact starting. An internal incident was declared 4 minutes thereafter. However, the root cause…
The programming language's maintenance strategy has been updated to deliver both security patches and select backported bug resolutions going forward. Because of this expanded focus on stability, the development team has officially rebranded the support period as the legacy maintenance phase. This ongoing support remains fully guaranteed through the previously established timeline ending in March 2027.
The core language maintainers have delivered critical maintenance updates addressing identified security vulnerabilities and routine software flaws. Published by k0kubun, this distribution arrives alongside coordinated maintenance releases across multiple supported release lines to ensure ecosystem stability. Developers relying on these platforms can now access updated packages through standard project distribution channels.
Maintainers have delivered vital security patches and general bugfixes through community contributions from numerous developers and testers. This deployment addresses ongoing stability concerns for systems running modern branches of the language. It arrived as part of a synchronized maintenance cycle impacting several supported release lines on the same day.