Previous incidents

The batches are now processing much faster. The issue was the default batch processing concurrency for an organization was too low. Organizations that were already on Hobby or Pro had higher concurrency but newer orgs didn't. This created a backlog.

We've increased the default processing concurrency for each org and will look at the upgrade path for paid customers as well.

We're monitoring the processing – it should get back to near realtime soon.

Runs are now syncing live and the dashboard is back to normal.

Fix has been applied and realtime streams v2 is fully operational.

Operations have returned to normal, we're continuing to investigate the root cause and will provide more detail as we know more.

The issue in ClickHouse is now resolved. The dashboard is back to being fully operational.

The root cause was a faulty node in the ClickHouse cluster which we couldn't kill. We're speaking to the ClickHouse Cloud team to find out why it happened.

Deployment for new projects have been fully restored.

The runs list has all caught up and the dashboard is no longer displaying stale data. We're continuing to investigate the root cause of this issue

Realtime updates back to normal. We're monitoring recovery.

We've published a full post-mortem on this incident here: https://trigger.dev/blog/clickhouse-too-many-parts-postmortem

We have recovered the clickhouse instance and the dashboard is responsive again and serving queries and data ingestion is back online. There has been some loss of otel data during this downtime but we don't know the extent of it at this moment as we continue to recover and investigate.

Realtime is working again. ElectricSQL was only working for the first 10 seconds after the server came up. We updated some settings around the startup sequencing that means it is now working properly and are working with the Electric team to determine why this was happening.