Runs have been dequeuing quickly for some time now, so marking this as resolved. We're continuing to monitor it closely.

Runs dequeued for the entire period but queue times were longer than normal, across all customers.

The vast majority of queues have reduced back to normal length already or will soon.

We suspect this was caused by an underlying Digital Ocean networking issue, that meant our Kubernetes control plane nodes were slow to create and delete pods. We are trying to figure out if there's anything in the short term we can do to reduce the likelihood of this happening again.

We are planning to move our primary US worker cluster to AWS, which is where we already host our database, dashboard, API, and all other services apart from the Worker cluster. This decision was made to increase reliability and improve response times.

Run queues are processing slower than normal due to an issue with our Kubernetes control planes.

We are investigating and will update this status as soon as we know more.