The issue is now resolved and DNS service is fully recovered. The degradation was caused by abnormal DNS load patterns, which caused intermittent issues with external resolvers.

We've already applied some changes to harden our networking infrastructure and will monitor closely to confirm stability.

This is a coredns issue.

We have spread out the coredns pods onto more nodes and added more replicas. This wasn't the same issue as we have seen with previous DNS problems. We're determining why this happened still.

We're monitoring the change as the error rate comes down.

This can impact HTTP requests including database queries. It seems to be impacting up to 5% of database queries for some users.