Previous incidents

Full service has been restored. Task execution is back to normal. If you experienced failures between 01:37 and 04:19 UTC, those runs can be retried successfully now.

What happened: During a period of high activity, a backlog of completed runs ...

The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal.

All schedules have been fully restored.

We have finally been able to provision additional capacity and logs are working again. A full post-mortem will follow.

The runs list is now up to do and syncing live updates again.

The new batch concurrency processing defaults have brought the processing queue down to zero

Runs are now syncing live and the dashboard is back to normal.

Fix has been applied and realtime streams v2 is fully operational.

Operations have returned to normal, we're continuing to investigate the root cause and will provide more detail as we know more.

The issue in ClickHouse is now resolved. The dashboard is back to being fully operational.

The root cause was a faulty node in the ClickHouse cluster which we couldn't kill. We're speaking to the ClickHouse Cloud team to find out why it happened.

Deployment for new projects have been fully restored.

The runs list has all caught up and the dashboard is no longer displaying stale data. We're continuing to investigate the root cause of this issue

Realtime updates back to normal. We're monitoring recovery.

We've published a full post-mortem on this incident here: https://trigger.dev/blog/clickhouse-too-many-parts-postmortem

We have recovered the clickhouse instance and the dashboard is responsive again and serving queries and data ingestion is back online. There has been some loss of otel data during this downtime but we don't know the extent of it at this moment as we continue to recover and investigate.

Realtime is working again. ElectricSQL was only working for the first 10 seconds after the server came up. We updated some settings around the startup sequencing that means it is now working properly and are working with the Electric team to determine why this was happening.