Previous incidents

v4 dequeue performance has now improved again, and we're working on two things:

A short term fix to prevent this from happening again in the short term, to be deployed today.

A long term fix for dequeue performance that will be worked on and hopefully shipped this week, which will vastly improve dequeue performance and scaling.

Queues are back to nominal length and have been for some time.

This issue was caused by a huge influx of queues, which meant we weren't considering them all when selecting queues for dequeuing.

We have increased some settings to make this better and we're looking at what we can do in the future to make this scale better for the next 10–100x multiple of queues.

Queues have been operating at full speed since 16:05 UTC.

We have found an edge case in the dequeue algorithm that can cause slower dequeue times. We're looking into a fix.

Runs have been dequeuing quickly for some time now, so marking this as resolved. We're continuing to monitor it closely.

Runs dequeued for the entire period but queue times were longer than normal, across all customers.

The vast majority of queues have reduced back to normal length already or will soon.

We suspect this was caused by an underlying Digital Ocean networking issue, that meant our Kubernetes control plane nodes were slow to create and delete pods. We are trying to figure out i...