{"id":37372,"date":"2019-10-31T22:17:16","date_gmt":"2019-10-31T19:17:16","guid":{"rendered":"https:\/\/prohoster.info\/blog\/etc-resolv-conf-dlya-kubernetes-pods-optsiya-ndots-5-kak-eto-mozhet-negativno-skazatsya-na-proizvoditelnosti-prilozheniya\/"},"modified":"2019-10-31T22:17:16","modified_gmt":"2019-10-31T19:17:16","slug":"etc-resolv-conf-dlya-kubernetes-pods-optsiya-ndots-5-kak-eto-mozhet-negativno-skazatsya-na-proizvoditelnosti-prilozheniya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/etc-resolv-conf-dlya-kubernetes-pods-optsiya-ndots-5-kak-eto-mozhet-negativno-skazatsya-na-proizvoditelnosti-prilozheniya","title":{"rendered":"\/etc\/resolv.conf \u0434\u043b\u044f Kubernetes pods, \u043e\u043f\u0446\u0438\u044f ndots:5, \u043a\u0430\u043a \u044d\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043d\u0435\u0433\u0430\u0442\u0438\u0432\u043d\u043e \u0441\u043a\u0430\u0437\u0430\u0442\u044c\u0441\u044f \u043d\u0430 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"\/etc\/resolv.conf<\/p>\n

<\/p>\n

Not long ago, we launched Kubernetes 1.9 on AWS using Kops. Yesterday, during a smooth rollout of new traffic on our largest Kubernetes clusters, I began noticing unusual DNS resolution errors logged by our application.<\/p>\n

<\/a><\/noindex><\/p>\n

This has been discussed on GitHub for quite a while, so I decided to look into it as well. Eventually, I figured out that in our case, it was caused by increased load on<\/a><\/noindex>. The most interesting and new aspect for me was the reason behind the significant increase in DNS query traffic. This, along with what to do about it, is the topic of my post. Modifying podNetwork<\/code> and dnsmasq<\/code>DNS resolution within a container\u2014just like in any Linux system\u2014is determined by the configuration file<\/p>\n

<\/p>\n

. By default, Kubernetes uses \/etc\/resolv.conf<\/code>ClusterFirst dnsPolicy<\/code> this , which means that any DNS query will be redirected to<\/code>, running in a pod dnsmasq<\/code>inside the cluster, which in turn redirects the request to the application Modifying podNetwork<\/code> , if the name ends with the cluster suffix; otherwise, it goes to a higher-level DNS server. Modifying podNetwork<\/code>Inside each container, the default configuration looks like this:<\/p>\n

<\/p>\n

File \/etc\/resolv.conf<\/code> nameserver 100.64.0.10\nsearch namespace.svc.cluster.local svc.cluster.local cluster.local \neu-west-1.compute.internal\noptions ndots:5<\/p>\n

<\/p>\n

As you can see, there are three directives:<\/code><\/pre>\n
<\/p>\n
The nameserver is the IP of the service.<\/p>\n
<\/p>\n
    \n
  1. Four local search domains are specified. Modifying podNetwork<\/code><\/li>\n
  2. There's an option : defines the search path for a specific domain. Interestingly,<\/code><\/li>\n
  3. ndots:5 An interesting part of this configuration is how local search domains and settings<\/code><\/li>\n<\/ol>\n
    <\/p>\n
    coexist. To understand this, you need to grasp how DNS resolution works for incomplete names. An interesting part of this configuration is how local search domains and settings<\/code> What is a fully qualified name?<\/p>\n
    <\/p>\n
    A fully qualified name is one for which local search will not be performed, and the name will be regarded as absolute during name resolution. By convention, DNS software considers a name fully qualified if it ends with a dot (.), and not fully qualified otherwise. That is,<\/h3>\n
    <\/p>\n
    is fully qualified, while google.com.<\/code> is not. mrkaran.dev<\/code> How is an incomplete name handled? <\/p>\n
    <\/p>\n
    When an application connects to a remote host indicated by a name, DNS name resolution is typically performed using a system call, such as<\/h3>\n
    <\/p>\n
    getaddrinfo() . However, if the name is incomplete (does not end with a .), it is interesting to see whether the system call attempts to resolve the name as absolute first or goes through local search domains first. This depends on the option<\/code>From the manual on ndots<\/code>.<\/p>\n
    <\/p>\n
    From the manual on contains the following information:<\/code>:<\/p>\n
    <\/p>\n
    ndots:n\n\nsets the threshold for the number of dots that must appear in a name before an initial absolute query is made. The default value for n is 1, meaning that if there are any dots in the name, the name will first be attempted as an absolute name before any search list elements are added.<\/code><\/pre>\n
    <\/p>\n
    This means that if ndots<\/code> a value of 5 is set and the name contains fewer than 5 dots, the system call will try to resolve it sequentially, first going through all local search domains, and upon failure, eventually resolve it as an absolute name.<\/p>\n
    <\/p>\n
    So why An interesting part of this configuration is how local search domains and settings<\/code> can it negatively impact application performance?<\/h3>\n
    <\/p>\n
    As you understand, if your application uses a lot of external traffic, for each established TCP connection (or rather, for each resolved name), it will issue 5 DNS queries before the name is correctly resolved, because it will first go through 4 local search domains and ultimately issue a resolution query for the absolute name.<\/p>\n
    <\/p>\n
    The following diagram shows the total traffic across our 3 kube-dns modules before and after we switched several hostnames configured in our application to fully qualified ones.<\/p>\n
    <\/p>\n
    \"\/etc\/resolv.conf<\/p>\n
    <\/p>\n
    The following diagram shows the application latency before and after we switched several hostnames configured in our application to fully qualified ones (the vertical blue line indicates deployment):<\/p>\n
    <\/p>\n
    \"\/etc\/resolv.conf<\/p>\n
    <\/p>\n
    Solution #1 \u2014 use fully qualified names<\/h3>\n
    <\/p>\n
    If you have few static external names (i.e., defined in the application configuration) to which you create a large number of connections, perhaps the simplest solution is to switch them to fully qualified by simply adding.<\/p>\n
    <\/p>\n
    This is not a final solution, but it helps quickly, albeit not elegantly, improve the situation. This patch we applied to resolve our issue, the results of which were shown in the screenshots above.<\/p>\n
    <\/p>\n
    Solution #2 \u2014 customization ndots<\/code> downward API support (simultaneously with this in dnsConfig<\/code><\/h3>\n
    <\/p>\n
    In Kubernetes 1.9, an alpha feature (beta version v1.10) appeared that allows better control over DNS parameters via pod properties in dnsConfig<\/code>. Among other things, it allows you to configure the value ndots<\/code> for a specific pod, i.e.<\/p>\n
    <\/p>\n
    apiVersion: v1\nkind: Pod\nmetadata:\n  namespace: default\n  name: dns-example\nspec:\n  containers:\n    - name: test\n      image: nginx\n  dnsConfig:\n    options:\n      - name: ndots\n        value: \"1\"<\/code><\/pre>\n
    <\/p>\n
    file \u2014 continuous reading of events from one or more local files;<\/h3>\n
    <\/p>\n