Previous incidents

Deployment for new projects have been fully restored.

The runs list has all caught up and the dashboard is no longer displaying stale data. We're continuing to investigate the root cause of this issue

Realtime updates back to normal. We're monitoring recovery.

We've published a full post-mortem on this incident here: https://trigger.dev/blog/clickhouse-too-many-parts-postmortem

We have recovered the clickhouse instance and the dashboard is responsive again and serving queries and data ingestion is back online. There has been some loss of otel data during this downtime but we don't know the extent of it at this moment as we continue to recover and investigate.

Realtime is working again. ElectricSQL was only working for the first 10 seconds after the server came up. We updated some settings around the startup sequencing that means it is now working properly and are working with the Electric team to determine why this was happening.