Some processing is slower than normal
Resolved
Sep 24 at 08:28pm BST
This issue is resolved, everything is back to normal.
This issue was caused by an exceptionally large number of v3 run alerts, caused by a run that was failing (from user code, not a Trigger.dev system problem). This caused us to hit Slack rate limits which slowed the processing down more.
We have scaled up the system that deals with this now so it should better deal with this. We've also changed the retrying settings for sending Slack alerts so it doesn't so aggressively retry.
Affected services
Trigger.dev API
Created
Sep 24 at 08:00pm BST
Impacted
- v2 run queues are processing slower than normal
- v3 alerts are taking longer than normal to send
- v3 scheduled runs may start slightly later than scheduled
- v3 triggerAndWait/batchTriggerAndWaits are taking longer than normal to continue their parent runs
We are working to get service back to normal.
Affected services
Trigger.dev API