Previous incidents

The ElectricSQL team created a fix for the Postgres transaction wraparound issue and we've confirmed that is now deployed and fixed on Prod.

We have moved all deploys to Europe on Depot as a temporary fix while they fix the underlying issue in the US. Deploys will be a bit slower than normal and the first one won't use the cache, but they should work.

Runs are processing normally again, queues should come down fast.

The Kubernetes database etcd didn’t allow new values. Increasing max sizes, restarting, and changing some other settings worked.

Realtime is back online.

We've made some configuration changes and have some more reliability fixes in progress to make this rock solid.

V2 queues are caught up. Now any queued runs are due to concurrency limits.

V3 was not impacted during the entire period.

We restarted all V2 worker servers and V2 runs started processing again. We are still investigating the underlying cause to prevent this happening again. There were no code changes or deploys during this period and the overall V2 load wasn't unusual.

Realtime is recovering after a restart and a clearing of the consumer cache, but the underlying issue has not been solved. We're still working on a fix and will update as we make progress.

The networking issues from our worker cluster cloud provider is no longer happening. Networking has been back to full speed for the past 10 minutes and run are processing fast.