Major outage

Incident Report for Asana

Resolved

Incident: From 2023-03-15 14:38 UTC to 2023-03-15 15:18 UTC there was a webhook outage for about 40 minutes in the US region. A bad deployment caused reduced capacity and delays in scheduling some asynchronous work, including webhook delivery.

Impact: Webhook users experienced a considerable amount of delay, max ~one hour difference between events generated and events delivered. There was no data loss due to this delay and backlog was recovered in ~20 minutes

Moving forward: We have added better tooling that would help us investigate this issue faster and have increased the priority of the job responsible for streaming webhook events.

Posted Mar 15, 2023 - 15:00 UTC