Postmortem: Missing task logs and traces

Date: 2026-09-03
Duration: ~67 minutes
Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dro...

We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes

A small percentage of streams experienced issues but it's resolved now.

The root cause was some server crashes for our provider which they're working on a permanent fix for now.