Partial outage of Dagster Cloud agent requests

Incident Report for Dagster Cloud

Resolved

We experienced a partial outage for 8 minutes between 2022-06-30 21:43 UTC and 21:51 UTC. During this window, some requests were not enqueued for agents to process. You may have experienced a missed schedule or sensor tick during this window.

During this window, we performed a rolling deploy of Dagster Cloud’s API servers. The newer version included a change to the agent request schema that was incompatible with older agent requests that had not yet been processed by an agent. Any request enqueued by a server running the older version and dequeued by a server running the newer version did not make it to your agent. All other requests were processed normally.

This incident is now resolved and steps are being taken to extend our existing backwards compatibility test suite to prevent this type of error going forward.

Posted Jun 30, 2022 - 21:43 UTC