Incident: A bug caused stale routing configuration for 49 hours, primarily in the US data center. We became aware of this through user reports, and cycling the affected servers restored functionality.

Impact: From 2024-03-02 11:40 UTC - 2024-03-04 12:00 UTC, up to 0.5% of users were unable to establish new sessions connecting to Asana.

Moving forward: We are adding monitoring for this failure mode, and have identified further improvements to detect problems like this. We have identified improvements in our routing layer configuration. We will be conducting a retrospective to explore how we can prevent similar issues in the future.

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.