Partial Outage
Incident Report for Asana
Postmortem

Incident: A change to how we load data from our databases led to increased server memory usage. During a period of heavy traffic the increased memory pressure exceeded a critical threshold.  This caused some servers to become overloaded, which resulted in slow or unresponsive request handling and retries. We reverted to a prior deployment and observed system health recover.

Impact: Between 13:47 and 15:20 UTC on 2024-03-07, attempts to create or edit data in Asana were delayed up to a few minutes, about 1% of web sessions crashed, and some API requests failed or were delayed. Background actions such as automations and email notifications were also delayed.

Moving forward: We have identified and reverted the problematic change and are improving our memory usage monitoring to identify regressions more quickly before causing user impact. Additionally, we discovered an issue with a safety measure to prevent excessive memory usage that we will address to prevent similar memory pressure issues in the future. 

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

Posted Mar 10, 2024 - 17:51 UTC

Resolved
This incident has been resolved.
Posted Mar 07, 2024 - 15:32 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 07, 2024 - 15:26 UTC
Update
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers and some changes to e.g. tasks may not be saving correctly. Our Development Team is currently investigating this issue and we aim to have this resolved as soon as possible. Our sincere apologies for the inconvenience; if you're impacted by this issue, please check this page for the latest updates.
Posted Mar 07, 2024 - 15:16 UTC
Investigating
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers. Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates.
Posted Mar 07, 2024 - 15:07 UTC
This incident affected: US (App, API, Mobile, Automations and Background Actions, Webhooks and Event Streams), EU (App, API, Mobile, Automations and Background Actions, Webhooks and Event Streams), Japan (App, API, Mobile, Automations and Background Actions, Webhooks and Event Streams), and Australia (App, API, Mobile, Automations and Background Actions, Webhooks and Event Streams).