Previous incidents

DNS service is now back to fully operational. Increased traffic combined with a routine infrastructure rollout caused intermittent DNS resolution failures. We've tuned our DNS configuration to resolve the issue and are working on longer-term improvements to prevent recurrence.

The issue has been resolved. Dashboard and telemetry are now fully operational.

The issue is now resolved and dequeue times are back to normal. Mainly free-tier runs were affected. This was caused by a spike in the free-tier run volume.

Full service has been restored. Task execution is back to normal. If you experienced failures between 01:37 and 04:19 UTC, those runs can be retried successfully now.

What happened: During a period of high activity, a backlog of completed runs ...

The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal.

All schedules have been fully restored.

We have finally been able to provision additional capacity and logs are working again. A full post-mortem will follow.

The runs list is now up to do and syncing live updates again.

The new batch concurrency processing defaults have brought the processing queue down to zero