Postmortem: Missing task logs and traces

Date: 2026-09-03
Duration: ~67 minutes
Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand.

What customers experienced

Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected.

Root cause

A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert.

The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss.

We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened.

What we've done

• Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors.
• Re-rolled the release the following morning with the column visible on every replica, which went out cleanly.
• Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire.

Ongoing

• Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page.
• A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed.
• A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped.
• Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause.

We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces.

The ingestion delay is down to normal levels.

A post-mortem will follow.

Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue.

We will resolve this issue with more details when the ingestion delays are back to normal.

We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data.

The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs.

This is being caused by higher load and slow auto-scaling in our ClickHouse cluster.