Uncached deploys are causing runs to be queued
Resolved
Mar 04 at 12:40pm GMT
We tracked this down to a broken deploy pipeline which reverted one of our internal components to a previous version. This caused a required environment variable to be ignored.
We have applied a hotfix and will be making more permanent changes to prevent this from happening again.
Affected services
Trigger.dev API
Updated
Mar 04 at 12:25pm GMT
All runs should be dequeuing again. Runs that were impacted by this will start dequeuing again. These runs were in the queue for around 30 mins.
This issue was caused by new servers not pulling the correct authentication details for pulling Docker images. We are trying to determine why this happened as we verified the env vars were set correctly but weren't being picked up.
We're going to continue actively monitoring this until we're happy that it's fully resolved.
Affected services
Trigger.dev API
Created
Mar 04 at 11:30am GMT
Some runs associated with new deploys are not processing from queued to executing.
This is an issue with pulling new deploys through the Docker image cache.
Deploys from before 11:30 UTC should not be impacted (unless they haven't done a run in the past few days).
We're working on a fix or this.
Affected services
Trigger.dev API