{"id":80532,"date":"2020-05-07T01:42:17","date_gmt":"2020-05-06T23:42:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem"},"modified":"2020-05-07T01:42:17","modified_gmt":"2020-05-06T23:42:17","slug":"problemy-s-dns-v-kubernetes-publichnyj-postmortem","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem","title":{"rendered":"DNS-i probleemid Kuberneteses. Avalik postmortem","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

M\u00e4rkus t\u00f5lkes:<\/b> see translation of the public postmortem from the engineering blog of the company Preply<\/a><\/noindex>. It describes the issue with conntrack in the Kubernetes cluster that led to partial downtime of some production services.<\/i><\/p>\n

This article may be useful for those who want to learn a bit more about postmortems or prevent some potential DNS issues in the future.<\/p>\n

\"DNS-i
\nThis is not DNS
\nIt cannot be that this is DNS
\nThis was DNS<\/i>
\n<\/a><\/noindex><\/p>\n

A bit about postmortems and processes at Preply<\/h2>\n

<\/p>\n

The postmortem describes a failure or any event in production. A postmortem includes a timeline of events, user impact description, root cause, actions, and lessons learned.<\/p>\n

Seeking SRE<\/a><\/noindex><\/i><\/p><\/blockquote>\n

\nAt weekly meetings with pizza, in the circle of the technical team, we share various information. One of the most important parts of such meetings are postmortems, which are usually accompanied by a slideshow presentation and a more in-depth analysis of the incident that occurred. Although we do not 'clap' after postmortems, we try to cultivate a 'blameless' culture (blameless culture<\/a><\/noindex>). We believe that writing and presenting postmortems can help us (and not only us) in preventing similar incidents in the future, which is why we share them.<\/p>\n

Individuals involved in the incident should feel that they can talk about it in detail without fear of punishment or retribution. No blame! Writing a postmortem is not punishment, but a learning opportunity for the entire company.<\/p>\n

Keep CALMS & DevOps: S is for Sharing<\/a><\/noindex><\/i><\/p><\/blockquote>\n

<\/p>\n

DNS issues in Kubernetes. Postmortem<\/h2>\n

\nKuup\u00e4ev:<\/b> 28.02.2020<\/p>\n

Autorid:<\/b> Amet U., Andrey S., Igor K., Alexey P.<\/p>\n

Olek:<\/b> Completed<\/p>\n

L\u00fchidalt:<\/b> Partial DNS unavailability (26 min) for some services in the Kubernetes cluster<\/p>\n

Impact:<\/b> 15,000 events lost for services A, B, and C<\/p>\n

Root Cause:<\/b> Kube-proxy was unable to correctly remove the old entry from the conntrack table, so some services were still trying to connect to non-existent pods.<\/p>\n

E0228 20:13:53.795782       1 proxier.go:610] Failed to delete kube-system\/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...<\/code><\/pre>\n
\nTrigger:<\/b> Due to low load inside the Kubernetes cluster, the CoreDNS autoscaler reduced the number of pods in the deployment from three to two.<\/p>\n
Lahendus:<\/b> J\u00e4rjekordne rakenduse juurutamine t\u00f5i kaasa uute s\u00f5lmede loomise, CoreDNS-autoscaler lisas klastrite teenindamiseks rohkem pude, mis p\u00f5hjustas conntrack tabeli \u00fclekirjutamise.<\/p>\n
Avastamine:<\/b> Prometheuse j\u00e4lgimine tuvastas suure hulga 5xx vigu teenustes A, B ja C ning algatas v\u00e4ljakutse valveinseneridele.<\/p>\n
\"DNS-i
\n5xx vead Kibanas<\/i><\/p>\n
Tegevused<\/h3>\n
<\/p>\n
Tegevus
\nT\u00fc\u00fcp
\nVastutav
\n\u00dclesanne<\/p>\n
Keela CoreDNS-i automaatne skaleerimine
\nennetama.
\nAmet U.
\nDEVOPS-695<\/p>\n
Paigalda vahem\u00e4lu DNS-server
\nv\u00e4hendama.
\nMax V.
\nDEVOPS-665<\/p>\n
Seadista conntrack j\u00e4lgimine
\nennetama.
\nAmet U.
\nDEVOPS-674<\/p>\n
<\/p>\n
\u00d5pitud \u00f5ppetunnid<\/h3>\n
\nMis l\u00e4ks h\u00e4sti:<\/b><\/p>\n