Previous incidents

Queue times are back to normal. We had an unprecedented number of new v4 runs. We've adjusted our autoscaling rules across multiple services to account for this. We are looking into how to avoid this happening as v4 scales up.

This was resolved at 00:29am. Logs and the API started recovering once a valid partition was in place.

The dashboard/runs.list is back to normal. We're working on and deploying multiple changes which will reduce and prevent these kind of issues from happening.

The runs list is now fully operational. There is still missing data that we will be backfilling ASAP.

We've opened a case with ClickHouse Cloud to try and understand why this happened.

Batches are processing as normal now. We have increased future capacity.

This was caused by a runaway loop of batches by a customer and this part of the system didn't have enough capacity to process them all fast enough.

We are updating how we process and rate-limit batches to prevent this from happening again, as well as improved internal alerts if similar issues happen in the future.

Realtime is sending updates again. The attached storage stopped working and restarting the AWS task didn't work. A hard reset caused it to become healthy again.

We're looking into how to prevent this from happening again

This is resolved – Realtime is sending updates again.

Restarting Electric released and reacquired the Postgres replication slot. We're discussing why this happened to try and prevent it in the future.

This has been resolved by rolling back to ElectricSQL 1.0.3.

We were on 1.0.20 which worked when we first deployed it yesterday. Then on a second deploy today it stopped receiving publications from Postgres. We tried restarting it a few times but it wouldn't receive any updates, so we rolled back.