Previous incidents
Intermittent DNS failures affecting some run executions
Resolved Jan 23 at 04:19am GMT
Full service has been restored. Task execution is back to normal. If you experienced failures between 01:37 and 04:19 UTC, those runs can be retried successfully now.
What happened: During a period of high activity, a backlog of completed runs ...
1 previous update
Dashboard runs list is delayed
Resolved Jan 21 at 05:32pm GMT
The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal.
2 previous updates
Some schedules have stopped working
Resolved Jan 21 at 11:15am GMT
All schedules have been fully restored.
1 previous update
Issue with task logs
Resolved Jan 16 at 05:44pm GMT
We have finally been able to provision additional capacity and logs are working again. A full post-mortem will follow.
1 previous update
Dashboard runs list is delayed
Resolved Jan 02 at 09:16pm GMT
The runs list is now up to do and syncing live updates again.
4 previous updates
Batches are slow to process
Resolved Jan 01 at 03:10pm GMT
The new batch concurrency processing defaults have brought the processing queue down to zero
2 previous updates