Postmortem -
Read details
Aug 31, 18:17 UTC
Resolved -
Impact: Dagster+ hybrid agents connecting to the US region via AWS PrivateLink experienced connection loss.
Duration: ~30 minutes (17:35 – 18:05 UTC / 13:35 – 14:05 ET on 2026-08-29)
Summary:
As part of a planned infrastructure migration, we rotated the network load balancers behind our PrivateLink endpoint service. This involved disconnecting the previous set of consumer VPC endpoints so that connections would be reopened on the new infrastructure. During the ~30 minute window between endpoint disconnection and full re-establishment, hybrid agents connecting via PrivateLink were unable to reach the Dagster+ control plane and appeared as unhealthy in our health dashboard.
Once new endpoints came online, agents automatically reconnected and returned to a healthy state without customer action.
What was affected:
Hybrid agents connecting to Dagster+ US region via AWS PrivateLink
Job runs, code location updates, and metadata operations dependent on that connection were paused (queued locally) until the connection recovered
What was NOT affected:
The Dagster+ UI and API remained fully available
Non-PrivateLink hybrid agents (using public network connectivity)
Dagster+ Serverless (EU and US)
Dagster+ EU hybrid customers
Timeline (EDT):
12:35 — Existing PrivateLink consumer endpoints intentionally disconnected as part of the migration
13:05 — Full recovery to pre-migration baseline
Root Cause:
Though this was a planned migration, provisioning the new endpoints took significantly longer than expected due to transient errors, leading to noticeable customer impact. This possibility should have been accounted for and communicated to customers in advance.
Aug 29, 17:30 UTC
