Back to overview
Degraded

Realtime is degraded

Dec 31 at 05:30pm GMT
Affected services
Realtime

Resolved
Jan 02 at 12:07pm GMT

The ElectricSQL team created a fix for the Postgres transaction wraparound issue and we've confirmed that is now deployed and fixed on Prod.

Updated
Jan 01 at 09:58pm GMT

We've narrowed down the exact cause of the issue that is happening in Electric and awaiting their engineer to get some sleep and submit a patch in the morning. Electric "subscribes" to updates in our database and compares the "transaction ID" or "xid" of the update in the postgresql WAL with the "current xmin". Our database has a "current xmin" which is very large (4.3b) and which is over the maximum transaction ID of 2^32. This is normal and is a feature of the transaction ID wraparound logic in postgresql. So new transactions have "wrapped around" back to 0 and now have lower transaction IDs that the current snapshot, and electric isn't modulo'ing the current snapshot xmin before they compare the transaction ID, so they are basically detecting all transactions as "old" even though they are newer.

Updated
Jan 01 at 03:58pm GMT

The very helpful and responsive engineers of Electric SQL (which powers Realtime) have possibly found a reason and are working on a reproduction. We're ready to deploy updates to our electric server as soon as those are made available to us. The short story is it's related to a transaction ID wraparound issue, but we'll update when we know more.

Updated
Jan 01 at 04:11am GMT

Nothing we have tried has fixed the issue. We're working with the electric team but unfortunately the timing is not great... most people are either sleeping or celebrating new years 🎉. We're hoping to be able to get this issue resolved ASAP given those constraints, and will update again soon.

Updated
Jan 01 at 12:23am GMT

We've exhausted our current runbook related to fixing Realtime issues and the service remains degraded. We're continuing to investigate and will update as soon as we are able to provide more information.

Created
Dec 31 at 05:30pm GMT

Some runs are not being updated by Realtime. All other systems are operating normally.