The ingestion delay is down to normal levels.

A post-mortem will follow.

Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue.

We will resolve this issue with more details when the ingestion delays are back to normal.

We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data.

The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs.

This is being caused by higher load and slow auto-scaling in our ClickHouse cluster.