Incident: On 2024-07-01 22:30 UTC, changes related to ongoing work towards smaller, independent deployment environments resulted in an invalid configuration affecting approximately 18% of our customers in our US datacenter. Resolution was delayed due to an unrelated issue with our infrastructure configuration management tools, but the impact was resolved for half of the affected users at 23:35 and for all affected users by 23:42.

Impact: Affected customers were unable to open our application (either with a web browser or with our desktop application). Existing sessions worked, and no data was lost. Mobile and API were unaffected. Users in other data centers (and ~82% of users in the US datacenter) were unaffected by this event.

Moving forward: We have done a retrospective analysis using the 5 Whys process, and have identified bugs, process improvements, and monitoring improvements to avoid future downtime while rolling out more fine-grained deployment environments.

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.