{"id":80532,"date":"2020-05-07T01:42:17","date_gmt":"2020-05-06T23:42:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem"},"modified":"2020-05-07T01:42:17","modified_gmt":"2020-05-06T23:42:17","slug":"problemy-s-dns-v-kubernetes-publichnyj-postmortem","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem","title":{"rendered":"DNS Issues in Kubernetes. Public Postmortem","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Nota:<\/b> this is a translation of a public postmortem from the engineering blog of the company Preply<\/a><\/noindex>. It describes a problem with conntrack in the Kubernetes cluster that led to partial downtime of some production services.<\/i><\/p>\n

This article may be useful for those who want to learn a little more about postmortems or prevent some potential DNS issues in the future.<\/p>\n

\"DNS
\nThis is not DNS
\nCannot be that this is DNS
\nThis was DNS<\/i>
\n<\/a><\/noindex><\/p>\n

A little about postmortems and processes at Preply<\/h2>\n

<\/p>\n

A postmortem describes a failure or any event in production. A postmortem includes a timeline of events, a description of the impact on users, root cause, actions taken, and lessons learned.<\/p>\n

Seeking SRE<\/a><\/noindex><\/i><\/p><\/blockquote>\n

\nAt weekly pizza meetings with the tech team, we share various information. One of the most important parts of these meetings is the postmortems, which are often accompanied by a presentation with slides and a deeper analysis of the incident that occurred. Although we do not \"applaud\" after postmortems, we strive to cultivate a culture of \"no blame\" (blameless culture<\/a><\/noindex>). We believe that writing and presenting postmortems can help us (and not just us) in preventing similar incidents in the future, which is why we share them.<\/p>\n

Those involved in the incident should feel that they can talk about it in detail without fear of punishment or retaliation. No blame! Writing a postmortem is not a punishment but a learning opportunity for the whole company.<\/p>\n

Keep CALMS & DevOps: S is for Sharing<\/a><\/noindex><\/i><\/p><\/blockquote>\n

<\/p>\n

DNS Issues in Kubernetes. Postmortem<\/h2>\n

\nDate:<\/b> 28.02.2020<\/p>\n

Authors:<\/b> Amet U., Andrey S., Igor K., Alexey P.<\/p>\n

Status:<\/b> Completed<\/p>\n

In short:<\/b> Partial DNS unavailability (26 min) for some services in the Kubernetes cluster<\/p>\n

Impact:<\/b> 15000 events lost for services A, B, and C<\/p>\n

Root cause:<\/b> Kube-proxy failed to correctly remove the old entry from the conntrack table, causing some services to still try to connect to non-existent pods.<\/p>\n

E0228 20:13:53.795782       1 proxier.go:610] Failed to delete kube-system\/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...<\/code><\/pre>\n
\nTrigger:<\/b> Due to low load within the Kubernetes cluster, CoreDNS-autoscaler reduced the number of pods in the deployment from three to two.<\/p>\n
Solution:<\/b> Another application deployment initiated the creation of new nodes, CoreDNS-autoscaler added more pods to serve the cluster, which prompted a rewrite of the conntrack table.<\/p>\n
Detection:<\/b> Prometheus monitoring detected a large number of 5xx errors for services A, B, and C, and initiated a call to on-duty engineers.<\/p>\n
\"DNS
\n5xx errors in Kibana<\/i><\/p>\n
Actions<\/h3>\n
<\/p>\n
Action
\nType
\nResponsible
\nTask<\/p>\n
Disable autoscaler for CoreDNS
\nprevent.
\nAmet U.
\nDEVOPS-695<\/p>\n
Set up a caching DNS server
\nreduce.
\nMax V.
\nDEVOPS-665<\/p>\n
Configure conntrack monitoring
\nprevent.
\nAmet U.
\nDEVOPS-674<\/p>\n
<\/p>\n
Lessons Learned<\/h3>\n
\nWhat went well:<\/b><\/p>\n