Deploys are failing with a 520 status code
Resolved
Jan 24 at 07:28pm GMT
Important: Upgrade to 3.3.12+ in order to deploy again
If you use npx you can upgrade the CLI and all of the packages by running:
npx trigger.dev@latest update
This should download 3.3.12 (or newer) of the CLI and then prompt you to update the other packages too.
If you have pinned a specific version (e.g. in GitHub Actions) you may need to manually update your package.json file or a workflow file.
Read our full package upgrading guide here: https://trigger.dev/docs/upgrading-packages
Affected services
Deployments
Updated
Jan 24 at 01:17pm GMT
The 520 errors are still happening with our Digital Ocean container registry, which is preventing the vast majority of deploys from working. We are speaking to their engineers but so far they haven't diagnosed a fix.
There have been no code changes from our side which caused this issue – it comes from a change somewhere in one of the third parties we use.
This is our deploy pipeline:
1. The Trigger.dev CLI runs the deploy command
2. Docker builds happen on Depot.dev
3. Depot pushes the image to our registry proxy (where we add registry credentials)
4. Our registry proxy pushes to Digital Ocean Container Registry (with the auth credentials)
From our logs we think the issue is that Digital Ocean have changed something that means they're now rejecting pushes they were accepting before.
We're working on two solutions in parallel:
1. Switching our container registry to something else, probably Docker Hub. This isn't super simple because we need to make it work with our proxy so we don't leak security credentials to users.
2. Stop using the proxy. This means generating temporary credentials that can only push to your project's repository. Those temporary tokens will be sent to the CLI, which means Depot.dev can push directly. AWS ECR supports doing this. Unfortunately this will require an updated CLI package to deploy…
We'll provide another update as soon as we have more information.
Affected services
Deployments
Updated
Jan 23 at 11:20pm GMT
We have confirmed that this is an issue pushing new images to our Digital Ocean Docker container registry. It's returning 520 errors.
We have submitted priority support tickets and are speaking to the Digital Ocean engineering teams about the problem. Currently we're waiting on them.
Affected services
Deployments
Created
Jan 23 at 09:10pm GMT
We're investigating why this is happening
Affected services
Deployments