Previous incidents

Postmortem: Intermittent DNS Resolution Failures (US region)

Date: 2026-04-21
Duration: ~1 hour
Impact: Intermittent DNS resolution failures for tasks running in our US region

Summary

Between approximately 15:50 and 16:50 UTC on 2026-04-21, some tasks running in our US region experienced intermittent DNS resolution failures when connecting to external services such as databases, object storage, and third-party APIs. The errors typically surfaced as EAI_AGAIN or transie...

The issue is now resolved and DNS service is fully recovered. The degradation was caused by abnormal DNS load patterns, which caused intermittent issues with external resolvers.

We've already applied some changes to harden our networking infrastructure and will monitor closely to confirm stability.

Realtime is back to live.

We're really sorry for this extended period of large delays. The service couldn't keep up the number of runs being processed and was falling further behind. We have made some configuration changes and upgraded it so it can cope with a higher throughput of runs.

If you were using our React hooks that just did streaming, they were unimpacted by this.

DNS service is now back to fully operational. Increased traffic combined with a routine infrastructure rollout caused intermittent DNS resolution failures. We've tuned our DNS configuration to resolve the issue and are working on longer-term improvements to prevent recurrence.

The issue has been resolved. Dashboard and telemetry are now fully operational.

The issue is now resolved and dequeue times are back to normal. Mainly free-tier runs were affected. This was caused by a spike in the free-tier run volume.