Queues are back to nominal length and have been for some time.

This issue was caused by a huge influx of queues, which meant we weren't considering them all when selecting queues for dequeuing.

We have increased some settings to make this better and we're looking at what we can do in the future to make this scale better for the next 10–100x multiple of queues.

We're dequeuing runs very fast again since the config update. Queues are coming down, mostly capped by the concurrency limits on queues.

We have identified the issue. A huge flood of new queues has caused an imbalance in the fair queue algorithm.

We're deploying a config env var change to the dequeuer now that should fix this. We're also figuring out how we can prevent this issue happening in the future if we hit another order of magnitude in the total queue count.

Version 3 queues aren't processing as fast as normal. Runs are still processing but not at the normal speed so some queues are getting longer.

We're investigating why this is happening and will provide updates.