Back to overview
Degraded

Uncached deploys are causing runs to be queued

Mar 04 at 11:30am GMT
Affected services
Trigger.dev API

Resolved
Mar 04 at 12:40pm GMT

We tracked this down to a broken deploy pipeline which reverted one of our internal components to a previous version. This caused a required environment variable to be ignored.

We have applied a hotfix and will be making more permanent changes to prevent this from happening again.

Updated
Mar 04 at 12:25pm GMT

All runs should be dequeuing again. Runs that were impacted by this will start dequeuing again. These runs were in the queue for around 30 mins.

This issue was caused by new servers not pulling the correct authentication details for pulling Docker images. We are trying to determine why this happened as we verified the env vars were set correctly but weren't being picked up.

We're going to continue actively monitoring this until we're happy that it's fully resolved.

Created
Mar 04 at 11:30am GMT

Some runs associated with new deploys are not processing from queued to executing.

This is an issue with pulling new deploys through the Docker image cache.

Deploys from before 11:30 UTC should not be impacted (unless they haven't done a run in the past few days).

We're working on a fix or this.