Previous incidents

We've opened a case with ClickHouse Cloud to try and understand why this happened.

Batches are processing as normal now. We have increased future capacity.

This was caused by a runaway loop of batches by a customer and this part of the system didn't have enough capacity to process them all fast enough.

We are updating how we process and rate-limit batches to prevent this from happening again, as well as improved internal alerts if similar issues happen in the future.

Realtime is sending updates again. The attached storage stopped working and restarting the AWS task didn't work. A hard reset caused it to become healthy again.

We're looking into how to prevent this from happening again

This is resolved – Realtime is sending updates again.

Restarting Electric released and reacquired the Postgres replication slot. We're discussing why this happened to try and prevent it in the future.

This has been resolved by rolling back to ElectricSQL 1.0.3.

We were on 1.0.20 which worked when we first deployed it yesterday. Then on a second deploy today it stopped receiving publications from Postgres. We tried restarting it a few times but it wouldn't receive any updates, so we rolled back.

v4 dequeue performance has now improved again, and we're working on two things:

A short term fix to prevent this from happening again in the short term, to be deployed today.

A long term fix for dequeue performance that will be worked on and hopefully shipped this week, which will vastly improve dequeue performance and scaling.

Queues are back to nominal length and have been for some time.

This issue was caused by a huge influx of queues, which meant we weren't considering them all when selecting queues for dequeuing.

We have increased some settings to make this better and we're looking at what we can do in the future to make this scale better for the next 10–100x multiple of queues.