US East 1 and EU Central 1 dequeue performance issues have been resolved. We're continuing to monitor the situation and a full post-mortem will follow.

US East 1 dequeue performance degradation has cleared but we're continuing to monitor the situation

US East 1 dequeues are currently degraded, as we work through the large backlog and deal with capacity issues with our upstream provider.

US East 1 continues to operate normally. EU Central 1 scheduled jobs are also back to normal operation. The remaining impact is EU on-demand workers still catching up on the backlog, so you may see delayed dequeuing of queued runs for a little while longer.

One thing to be aware of: if you have a large number of queued runs in EU Central 1, they can consume your concurrency and block runs in other regions (including US East 1) from executing. If your US East 1 runs aren't progressing despite that region being healthy, this is likely the cause. You can unblock yourself by cancelling the queued EU runs and replaying them in your normal region using the Bulk Action → Replay → Region override feature we just deployed.

The EU Central 1 region continues to drain and it looks like will be back to normal operation within the next hour at current drain rates. We have deployed the Bulk Action -> Replay -> Region override feature now which allows replaying runs into a different region. US East 1 continues to operate normally.

Our EU Central 1 is back online and processing the backlog of runs in that region as fast as we can. We will update again when we have an expected clearance time to get back to normal dequeue operations. We're also working on adding the ability to the Bulk Action feature to be able to replay runs in a different region from the replayed runs. US East 1 continues to operate normally. Full postmortem to follow

Our advice remains the same: switch your default region to us-east-1 if you can. More details below.

us-east-1
Operating normally with global queue times better than normal thresholds. If you have excessive runs queued in us-east-1:
1. Check your concurrency limits, increase as needed including at the account level from the Concurrency page (let us know via the in-app Help and Feedback and we will credit your account)
2. If you were on eu-central-1 they could be blocking your us-east-1 runs. Go to the Runs page and filter for "Queued" status and "eu-central-1" region. Then use the Bulk action on the right to cancel those runs.
3. When eu-central-1 is back you can replay these runs there.

eu-central-1
The cluster is being spun back up with more resources and the same settings we used to fix us-east-1. Currently no runs are executing here.

The same thundering herd issue that happened in us-east-1 caused this issue as everyone moved their traffic over. This cluster was smaller so it took less to cause this problem.

--

After this incident is over we will do a full post mortem with what happened along with a long list of things we need to do better. I've been noting them down and we've recorded this entire process.

We will also be offering credits to all impacted customers. I know this doesn't cover the cost to your businesses.

This is far from the bar we set for reliability. Sorry everyone,
Matt

us-east-1 is almost completely recovered although still with slower dequeue times than normal. We're continuing to investigate how to bring eu-central-1 back online.

As our us-east-1 region is recovering the eu-central-1 region is now experiencing the same degradation that occurred last night to us-east-1. We're continuing to investigate and attempting to bring the dequeues back online for eu-central-1, while also working through the us-east-1 backlog, which is continuing to clear but won't be fully clear for 1+ hr at the current rate as we battle to get server capacity from our upstream provider.

We've scaled up our services to handle the increased workload, and throughput is improving. There's still a significant backlog of runs to work through, so you may continue to see delays while we catch up. We're monitoring closely and processing through the queue as fast as we safely can.

EU Central 1 remains fully operational, so you can continue running workloads there in the meantime.

We are now processing a high number of us-east-1 runs, but there is a large backlog to get through. The backlog is reducing and we're looking to see if we can do this faster in a controlled way.

EU Central 1 remains fully operational, so you can continue running workloads there in the meantime.

The control plane has stabilized and core services are coming back online. We’re carefully restoring capacity in a controlled way.

EU Central 1 remains fully operational, so you can continue running workloads there in the meantime. No action needed on your end.

We’re close to full resolution and will share our next update shortly.

For users who have switched from the us-east-1 region to the eu-central-1 region, and are seeing slow to minimal run executions for newly created runs, one way to unlock these new run executions is to cancel any of your us-east-1 queued runs, by using the Region filter in the runs list and selecting the Bulk Action -> Cancel option. This us-east-1 outage has discovered a bug in our concurrency accounting that can cause queued runs in degraded regions to be counted against your concurrency limit.

We're continuing to make progress on restoring the us-east-1 region, and will update again shortly.

Just a note about switching your default region to the eu-central-1 cluster, and the behavior of how the switching of regions effects existing queued runs: already enqueued runs keep their original region (which is set at trigger time). Switching your default region effects new runs only, immediately.

It is possible to replay an individual runs on a different region, so one temporary workaround:

• Cancel existing queued runs in the us-east-1 region (using the Region filter)
• Go through each run cancelled individually and click through to the run detail page, and find the "Replay run" button in the top right
• Select eu-central-1 from the drop down

We're working on being able to streamline this process, and again we apologize for this outage.

We are continuing to investigate the issue and with us-east-1 but have no hard timeline on when we might be able to recover. The root cause of the issue is still not clear to us, and the nature of the issue is making it very difficult to ascertain the underlying cause. We're focusing on mitigation. eu-central-1 continues to operate under normal conditions.

Runs have stopped executing in us-east-1, we're investigating. The eu-central-1 region is still executing runs, consider switching to that region as a temporary workaround.

We managed to secure additional capacity, but large machines still disproportionately affected.

This predominantly affects larger machine sizes. AWS is having capacity issues.