The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal.

The dashboard runs list is now live again but some v3 logs are being dropped because of ingestion problems. We're investigating and will provide an update as soon as we have anything

Our run sync to clickhouse process is currently delayed. The runs list in the dashboard will be behind but runs are executing as normal.