Major outage EUDC
Incident Report for Asana
Postmortem

Incident: An off-peak operational task was suspended part-way through due to an error in our database upgrade script. This left our system in a configuration where caching was unavailable for some components, resulting in degraded performance during regional peak traffic. Additional (unrelated) errors in the Amazon Web Services availability zone for the region increased time to resolution. The net effect was that all EU traffic was unintentionally slowed down for approximately 10 hours. 

Impact: At times Asana was largely unavailable to new logins, with only existing sessions able to be used. Some of these existing sessions experienced degraded performance. For a period of time Asana was fully inaccessible. No customer data was lost.

Moving forward: This incident involved interaction across teams and systems. Our 5 Whys analysis of this event identified operational, organizational, and service changes to reduce the likelihood of incidents -- and decrease the time to resolution. We've recently staffed a Production Engineering team whose goal is to improve processes and systems across teams.

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

Posted Aug 23, 2021 - 18:02 UTC

Resolved
This incident has been resolved.
Posted Jul 13, 2021 - 17:27 UTC
Identified
We believe we have identified the underlying cause and we are taking action to mitigate the incident.
Posted Jul 13, 2021 - 15:38 UTC
Investigating
We are currently investigating this issue.
Posted Jul 13, 2021 - 15:08 UTC
Update
The AWS issue has been mitigated, but we’re now investigating a significant performance regression impacting our EU Data Center.
Posted Jul 13, 2021 - 14:53 UTC
Identified
Root cause identified as an upstream AWS incident affecting only our EU cluster. We appreciate your understanding as we work on a fix.
Posted Jul 13, 2021 - 13:11 UTC
Investigating
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers who are hosted in our EU Data Center. We are currently investigating this issue and hope to have it fixed as soon as possible. Our sincere apologies for the inconvenience; if you’re impacted by this issue, please keep an eye on this page for the latest updates.
Posted Jul 13, 2021 - 12:26 UTC
This incident affected: US (App).