Runs have been executing at normal speeds and queues down to normal size for a couple of hours so this is marked as resolved.

During this incident queue times were longer than normal for v2 runs. We've made some minor changes as well as increasing capacity. We are working on a larger change that we think should mean very large v2 run spikes don't cause these problems that should ship in the next few days.

v2 queues are getting smaller quickly now. v3 is still operating normally.

More capacity is now online and job queues are catching up, but not all the way. We're currently assessing whether it's possible to increase capacity even more.

We are currently struggling to keep up with v2 job capacity and there are some jobs are have been backed up. We're working on increasing the capacity now.