Incident: On June 28, 2024 between 0334 and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out.
Impact: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates.
Moving forward: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.