Previous incidents

Queue processing performance is back to normal because there's been a reduction in demand.

We have identified the underlying bottleneck and are working on a permanent fix. This shouldn't be a major change and should be live. There is a high degree of contention on an update when a single queue's concurrencyLimit is different on every call to trigger a task. This is an edge case we haven't seen anyone do before.

Important: Upgrade to 3.3.12+ in order to deploy again

If you use npx you can upgrade the CLI and all of the packages by running:
npx trigger.dev@latest update

This should download 3.3.12 (or newer) of the CLI and then prompt you to update the other packages too.

If you have pinned a specific version (e.g. in GitHub Actions) you may need to manually update your package.json file or a workflow file.

Read our full package upgrading guide here: https://trigger.dev/docs/upgrading-packages

The ElectricSQL team created a fix for the Postgres transaction wraparound issue and we've confirmed that is now deployed and fixed on Prod.

We have moved all deploys to Europe on Depot as a temporary fix while they fix the underlying issue in the US. Deploys will be a bit slower than normal and the first one won't use the cache, but they should work.

Runs are processing normally again, queues should come down fast.

The Kubernetes database etcd didn’t allow new values. Increasing max sizes, restarting, and changing some other settings worked.

Realtime is back online.

We've made some configuration changes and have some more reliability fixes in progress to make this rock solid.

V2 queues are caught up. Now any queued runs are due to concurrency limits.

V3 was not impacted during the entire period.

We restarted all V2 worker servers and V2 runs started processing again. We are still investigating the underlying cause to prevent this happening again. There were no code changes or deploys during this period and the overall V2 load wasn't unusual.