We tracked this down to a broken deploy pipeline which reverted one of our internal components to a previous version. This caused a required environment variable to be ignored.

We have applied a hotfix and will be making more permanent changes to prevent this from happening again.

All runs should be dequeuing again. Runs that were impacted by this will start dequeuing again. These runs were in the queue for around 30 mins.

This issue was caused by new servers not pulling the correct authentication details for pulling Docker images. We are trying to determine why this happened as we verified the env vars were set correctly but weren't being picked up.

We're going to continue actively monitoring this until we're happy that it's fully resolved.

Some runs associated with new deploys are not processing from queued to executing.

This is an issue with pulling new deploys through the Docker image cache.

Deploys from before 11:30 UTC should not be impacted (unless they haven't done a run in the past few days).

We're working on a fix or this.