Realtime is degraded
Resolved
Jan 02 at 12:07pm GMT
The ElectricSQL team created a fix for the Postgres transaction wraparound issue and we've confirmed that is now deployed and fixed on Prod.
Affected services
Realtime
Updated
Jan 01 at 09:58pm GMT
We've narrowed down the exact cause of the issue that is happening in Electric and awaiting their engineer to get some sleep and submit a patch in the morning. Electric "subscribes" to updates in our database and compares the "transaction ID" or "xid" of the update in the postgresql WAL with the "current xmin". Our database has a "current xmin" which is very large (4.3b) and which is over the maximum transaction ID of 2^32. This is normal and is a feature of the transaction ID wraparound logic in postgresql. So new transactions have "wrapped around" back to 0 and now have lower transaction IDs that the current snapshot, and electric isn't modulo'ing the current snapshot xmin before they compare the transaction ID, so they are basically detecting all transactions as "old" even though they are newer.
Affected services
Realtime
Updated
Jan 01 at 03:58pm GMT
The very helpful and responsive engineers of Electric SQL (which powers Realtime) have possibly found a reason and are working on a reproduction. We're ready to deploy updates to our electric server as soon as those are made available to us. The short story is it's related to a transaction ID wraparound issue, but we'll update when we know more.
Affected services
Realtime
Updated
Jan 01 at 04:11am GMT
Nothing we have tried has fixed the issue. We're working with the electric team but unfortunately the timing is not great... most people are either sleeping or celebrating new years 🎉. We're hoping to be able to get this issue resolved ASAP given those constraints, and will update again soon.
Affected services
Realtime
Updated
Jan 01 at 12:23am GMT
We've exhausted our current runbook related to fixing Realtime issues and the service remains degraded. We're continuing to investigate and will update as soon as we are able to provide more information.
Affected services
Realtime
Created
Dec 31 at 05:30pm GMT
Some runs are not being updated by Realtime. All other systems are operating normally.
Affected services
Realtime