Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error

Incident Report for Dagster Cloud

Resolved

This incident has been resolved.

Posted Sep 06, 2023 - 23:59 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 06, 2023 - 22:39 UTC

Identified

The issue has been identified and we are in the process of testing a fix.

Posted Sep 06, 2023 - 20:08 UTC

Investigating

We have observed an increase in timeouts from our storage engine during run execution, resulting in some runs failing with an error like the following in the event log:

dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: XXX)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}]

We are currently investigating the issue.

Posted Sep 06, 2023 - 19:30 UTC

This incident affected: API.