DNS Lookup in Kubernetes

Note. transl.: DNS problem in Kubernetes, or rather, parameter settings ndots, - surprisingly popular, and already Not the first year. In another note on this topic, its author, a DevOps engineer from a large brokerage company in India, tells in a very simple and concise manner what is useful to know for colleagues operating Kubernetes.

One of the main benefits of deploying applications on Kubernetes is seamless application discovery. Inter-cluster interaction is greatly simplified due to the service concept (Service), which is a virtual IP that supports a set of pod IP addresses. For example, if the service vanilla wants to contact the service chocolate, it can directly access the virtual IP for chocolate. The question arises: who in this case will resolve the DNS query to chocolate And How?

DNS name resolution is configured on the Kubernetes cluster using CoreDNS. Kubelet registers pod with CoreDNS as nameserver in files /etc/resolv.conf all pods. If you look at the content /etc/resolv.conf any pod, it will look something like this:

search hello.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.152.183.10
options ndots:5

This configuration is used by DNS clients to redirect queries to the DNS server. In file resolv.conf contains the following information:

• nameservers: Server to which DNS queries will be sent. In our case, this is the address of the CoreDNS service;
• search: Specifies the search path for a specific domain. It's curious that google.com or mrkaran.dev are not FQDNs (full domain names). According to the standard convention followed by most DNS resolvers, only those domains that end with a dot "." representing the root zone are considered full (FDQN) domains. Some resolvers are able to add a dot themselves. Thus, mrkaran.dev. is the fully qualified domain name (FQDN), and mrkaran.dev - not;
• ndots: The most interesting parameter (this article is about it). ndots sets the threshold number of dots in a query name before it is considered a "fully qualified" domain name. We'll talk more about this later when we analyze the DNS lookup sequence.

Let's see what happens when we request mrkaran.dev in pod:

$ nslookup mrkaran.dev
Server: 10.152.183.10
Address: 10.152.183.10#53

Non-authoritative answer:
Name: mrkaran.dev
Address: 157.230.35.153
Name: mrkaran.dev
Address: 2400:6180:0:d1::519:6001

For this experiment, I set the CoreDNS logging level to all (which makes it quite verbose). Let's look at the pod's logs coredns:

[INFO] 10.1.28.1:35998 - 11131 "A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000263728s
[INFO] 10.1.28.1:34040 - 36853 "A IN mrkaran.dev.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000214201s
[INFO] 10.1.28.1:33468 - 29482 "A IN mrkaran.dev.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000156107s
[INFO] 10.1.28.1:58471 - 45814 "A IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 56 0.110263459s
[INFO] 10.1.28.1:54800 - 2463 "AAAA IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 68 0.145091744s

Phew. Two things draw attention here:

• The request goes through all stages of the search until the answer contains the code NOERROR (DNS clients understand it and store it as a result). NXDOMAIN means that no record was found for the given domain name. Because the mrkaran.dev is not an FQDN (according to ndots=5), the resolver looks at the search path and determines the order of requests;
• Entries А и АААА act in parallel. The fact is that one-time requests in /etc/resolv.conf are configured by default to perform parallel lookups on both IPv4 and IPv6. You can override this behavior by adding the option single-request в resolv.conf.

Note: glibc can be configured to send these requests sequentially, and musl No, so Alpine users should take note.

Experimenting with ndots

Let's experiment a little more with ndots and see how this parameter behaves. The idea is simple: ndots determines whether the DNS client will treat the domain as absolute or relative. For example, in the case of a simple google, how does the DNS client know if this domain is absolute? If you set ndots equal to 1, the client will say: “Oh, in google there is no dot; Perhaps I'll go over the entire search list. However, if requested google.com, the suffix list will be completely ignored because the requested name satisfies the threshold ndots (there is at least one point).

Let's check it out:

$ cat /etc/resolv.conf
options ndots:1
$ nslookup mrkaran
Server: 10.152.183.10
Address: 10.152.183.10#53

** server can't find mrkaran: NXDOMAIN

CoreDNS logs:

[INFO] 10.1.28.1:52495 - 2606 "A IN mrkaran.hello.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000524939s
[INFO] 10.1.28.1:59287 - 57522 "A IN mrkaran.svc.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000368277s
[INFO] 10.1.28.1:53086 - 4863 "A IN mrkaran.cluster.local. udp 39 false 512" NXDOMAIN qr,aa,rd 132 0.000355344s
[INFO] 10.1.28.1:56863 - 41678 "A IN mrkaran. udp 25 false 512" NXDOMAIN qr,rd,ra 100 0.034629206s

Since in mrkaran there is not a single dot, the search was carried out through the entire list of suffixes.

Note: in practice the maximum value ndots limited to 15; by default in Kubernetes it is 5.

Application in production

If an application makes a lot of external network calls, DNS can become a bottleneck in case of heavy traffic, because many extra requests are made during name resolution (before the system gets to the right one). Applications don't usually add a root zone to domain names, but this is quite hackish. That is, instead of asking api.twitter.com, you can hardcode api.twitter.com. (with a dot) in the application, which will prompt DNS clients to perform authoritative lookups immediately on the absolute domain.

In addition, since Kubernetes 1.14, extensions dnsConfig и dnsPolicy received stable status. Thus, when deploying a pod, you can reduce the value ndots, say, up to 3 (and even up to 1!). Because of this, every message within a node will need to include the full domain. This is one of the classic trade-offs between performance and portability. I think this is only worth worrying about if ultra-low latency is vital to your application, since DNS results are also cached internally.

references

I first learned about this feature on K8s-meetup'eheld on January 25th. There was also talk about this issue.

Here are some links for further study:

• Explanationwhy ndots=5 in Kubernetes;
• Great stuff how changing ndots affects application performance;
• Discrepancies between musl and glibc resolvers.

Note: I chose not to use dig in this article. dig automatically adds a dot (root zone identifier), making the domain "full" (FQDN), not running it through the search list beforehand. Wrote about it in one of the previous publications. However, it is rather surprising that, in general, you have to set a separate flag for the standard behavior.

Good DNS'ing! See you later!

PS from translator

Read also on our blog:

• «Calico for networking in Kubernetes: an introduction and a bit of experience";
• «CoreDNS - DNS server for the cloud native world and Service Discovery for Kubernetes";
• "An Illustrated Guide to Networking in Kubernetes": parts 1 and 2 (network model, overlay networks), part 3 (services and traffic processing).

Source: habr.com