Incident: At 2025-02-06 15:37 UTC, a configuration change caused some of our servers to crash, leading to complete downtime for 20 minutes. We rolled back the configuration, but this event triggered a cascade failure in network routing components; our systems recovered over the next 30 minutes, with full recovery by 2025-02-06 16:30.
Impact: Users were unable to use Asana during this event in all regions, with downtime including App, API, and Mobile. No customer data was lost.
Moving forward: We have added validation to prevent a repeat of this issue, and are updating our networking components to avoid this form of cascade failure. We will be moving to staged configuration rollouts to identify this class of issue without causing downtime for customer traffic.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.