Previous incidents

We have recovered the clickhouse instance and the dashboard is responsive again and serving queries and data ingestion is back online. There has been some loss of otel data during this downtime but we don't know the extent of it at this moment as we continue to recover and investigate.

Realtime is working again. ElectricSQL was only working for the first 10 seconds after the server came up. We updated some settings around the startup sequencing that means it is now working properly and are working with the Electric team to determine why this was happening.

Depot resolved the issue on their side. Deployments are now working normally again.

The issue is now resolved. We scaled up massively to absorb a load spike from one of our customers. Mainly runs on large machines where impacted.

We are still working with clickhouse cloud on identifying the root cause of the issue but have pushed out a fix that has allowed the runs list to come back online and show live results in the meantime. We suspect the issue is related to our clickhouse server being "rotated" into another AZ which caused network degradation and cause our replication pipeline from postgresql to clickhouse to fall behind and unable to keep up with changes. The fix was to stop sending payload data to clickhouse wh...

Our remote build provider Depot.dev was able to fully recover after ongoing EC2 instance issues in us-east-1 - AWS had several regressions and limited service availability throughout the day.

Our clickhouse instance is back online and serving queries. The runs list is now working but some requests to clickhouse syncing run state are queued and waiting to finish. We should be all caught up in about 5-10 minutes.

Realtime is now back to normal operation, serving live updates. We'll continue to monitor the situation.

AWS us-east-1's latest update has stated that they are seeing "significant signs of recovery" and we're seeing the same in our worker cluster, as now image pulls are working and both cold and warm starts are now processing as normal. We'll continue to closely monitor the situation.

Runs on large-2x machines were processing more slowly due to incredibly high volume. One of our customers caused an infinite loop in their large machine tasks which kept triggering more runs. Runs on small and medium machines were unlikely to have been impacted by this.

This resolved shortly after a problematic control plane rollout.

Service in us-east-1 has fully recovered

Larger machines are now dequeuing faster.

We are going to change the packing algorithm we use which means this will be much less likely to happen in the future. That work will begin this week.

The database load is back to normal.

We're still investigating why this happened, the top theory at the moment is a an auto-vacuuming issue possibly to do with transaction wraparound.

This issue was caused by our us-nyc-3 cloud provider taking an abnormally long time to spin up new servers and capacity issues, along with some runs getting stuck after restoring from a snapshot and not completing in under 2 minutes, which also mostly happened in us-nyc-3.

We were seeing crashes on multiple servers in the EU only. It was related to an out of memory issue in our “supervisor” which meant some servers weren’t dequeuing consistently. We’ve changed some settings to allow for more memory. We're still investigating why these supervisor processes were using up too much memory and crashing, and we're monitoring the situation. us-east-1 runs have not been impacted.