The dashboard/runs.list is back to normal. We're working on and deploying multiple changes which will reduce and prevent these kind of issues from happening.

We're starting to see spikes in runs list performance again. This impacts the dashboard Tasks page, Runs page, and runs.list() SDK function.

Our best theory right now is that this is when ClickHouse Cloud cycles replica servers. We are working with their support to get to the bottom of this because CPU and memory are not high before this happens.

We have also made some changes since yesterday, as well as working on some mitigations (like falling back to Postgres). There are some dashboard changes going live soon where instead of it completely failing to load we show better error states.

Sorry about this everyone, we're working hard to prevent this from happening.