Previous incidents

February 2025
Feb 09, 2025
1 incident

Slow queue times and some runs system failing

Degraded

Resolved Feb 09 at 06:10pm GMT

Queue times and timeout errors are back to their previous levels.

Note that start times of containers are still slower than they should be, especially if you aren't doing a lot of runs.

There's a GitHub issue here about slow start times and what we're doing to make them consistently fast: https://github.com/triggerdotdev/trigger.dev/issues/1685

It will start with a new Docker image caching layer that will ship tomorrow.

What's causing these problems

We've had a more than 10x increase...

1 previous update

January 2025
Jan 30, 2025
1 incident

Slow queue times

Resolved Jan 30 at 10:10am GMT

Queue processing performance is back to normal because there's been a reduction in demand.

We have identified the underlying bottleneck and are working on a permanent fix. This shouldn't be a major change and should be live. There is a high degree of contention on an update when a single queue's concurrencyLimit is different on every call to trigger a task. This is an edge case we haven't seen anyone do before.

Jan 23, 2025
1 incident

Deploys are failing with a 520 status code

Downtime

Resolved Jan 24 at 07:28pm GMT

Important: Upgrade to 3.3.12+ in order to deploy again

If you use npx you can upgrade the CLI and all of the packages by running:
npx trigger.dev@latest update

This should download 3.3.12 (or newer) of the CLI and then prompt you to update the other packages too.

If you have pinned a specific version (e.g. in GitHub Actions) you may need to manually update your package.json file or a workflow file.

Read our full package upgrading guide here: https://trigger.dev/docs/upgrading-packages

3 previous updates

December 2024
Dec 31, 2024
1 incident

Realtime is degraded

Degraded

Resolved Jan 02 at 12:07pm GMT

The ElectricSQL team created a fix for the Postgres transaction wraparound issue and we've confirmed that is now deployed and fixed on Prod.

5 previous updates

Dec 02, 2024
1 incident

Deploys are failing due to a downstream provider

Degraded

Resolved Dec 02 at 10:09pm GMT

We have moved all deploys to Europe on Depot as a temporary fix while they fix the underlying issue in the US. Deploys will be a bit slower than normal and the first one won't use the cache, but they should work.

1 previous update