We stopped the issue at 15:10 UTC and no environment variables have been cleared since. A permanent fix is in progress. Variables that were cleared between 12:21 and 15:10 UTC are still empty and will not come back on redeploy. We have contacted every affected customer directly. Re-entering them in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. We're still working with affected customers to restore their values, so if you need help or have questio...

Logs are being fully ingested again.

The issue was with the internal workers in our shared OpenTelemetry service. They became unhealthy but weren't rolled. We're looking into why and how to prevent this from happening again.

Postmortem: Missing task logs and traces

Date: 2026-09-03
Duration: ~67 minutes
Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dro...

We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes

A small percentage of streams experienced issues but it's resolved now.

The root cause was some server crashes for our provider which they're working on a permanent fix for now.