Previous incidents
The v3 cluster is slow to accept new v3 tasks
Resolved May 14 at 08:00am BST
Runs are operating at normal speed again
There were pods in our cluster in the RunContainerError state, this happens when a run isn’t heartbeating back to the platform. We’re closely monitoring and have cleaned these. We’re determining which tasks caused this and what we can do to prevent this from happening in the future.
1 previous update
Queues and runs have been processing at good speeds now for several hours on ...
Resolved May 09 at 10:00pm BST
Before we get into what happened I want to emphasise how important reliability is to us. This has fallen very short of providing you all with a great experience and a reliable service. We're really sorry for the problems this has caused for you all.
All paying customers will get a full refund for the entirety of May.
What caused this?
This issue started with a huge number of queued events in v2. Our internal system that handles concurrency limits on v2 was slowing down the pro...