Just a note about switching your default region to the eu-central-1 cluster, and the behavior of how the switching of regions effects existing queued runs: already enqueued runs keep their original region (which is set at trigger time). Switching your default region effects new runs only, immediately.

It is possible to replay an individual runs on a different region, so one temporary workaround:

• Cancel existing queued runs in the us-east-1 region (using the Region filter)
• Go through each run cancelled individually and click through to the run detail page, and find the "Replay run" button in the top right
• Select eu-central-1 from the drop down

We're working on being able to streamline this process, and again we apologize for this outage.

We are continuing to investigate the issue and with us-east-1 but have no hard timeline on when we might be able to recover. The root cause of the issue is still not clear to us, and the nature of the issue is making it very difficult to ascertain the underlying cause. We're focusing on mitigation. eu-central-1 continues to operate under normal conditions.

Runs have stopped executing in us-east-1, we're investigating. The eu-central-1 region is still executing runs, consider switching to that region as a temporary workaround.

We managed to secure additional capacity, but large machines still disproportionately affected.

This predominantly affects larger machine sizes. AWS is having capacity issues.