The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal.

All schedules have been fully restored.

We have finally been able to provision additional capacity and logs are working again. A full post-mortem will follow.