Incident:

An experimental flag was enabled that changed initial page loading behavior. With the flag enabled, users were inadvertently directed to load a code version that was not servable by all groups of servers. Subsequent requests would fail when requesting this incompatible version, including requests to report the page load errors. 

‌

Impact: 

From 16:52 to 18:59 UTC, 10% of users were unable to load the Asana application in the desktop app or web browser. Existing sessions, as well as mobile and API traffic, were not impacted.

‌

Moving forward:

We reverted the configuration change and identified gaps in our monitoring. We are adding monitoring and alerting for version mismatches during page load in order to more quickly identify similar regressions in the future.

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.