🥇One of the features of Chromium places a huge load on root DNS servers

The Chromium browser, an actively developing open-source parent of Google Chrome and the new Microsoft Edge, has attracted serious negative attention due to a feature intended with good intentions: it checks whether the user's provider is "stealing" nonexistent query results from domains.

Intranet Redirect Detector, generating fake requests for random "domains" whose existence is statistically improbable, accounts for about half of the total traffic received by root DNS servers worldwide. Verisign engineer Matt Thomas wrote a lengthy post post on the APNIC blog detailing the problem and assessing its scale.

How DNS resolution is typically performed

These servers are the ultimate authority to consult for resolving .com, .net, and so on, to inform you that frglxrtmpuf is not a top-level domain (TLD).

DNS, or Domain Name System, is the system that allows computers to convert memorable domain names like arstechnica.com into far less convenient IP addresses, such as 3.128.236.93. Without DNS, the Internet could not exist in a user-friendly form, and so the unnecessary burden on top-level infrastructure is a real issue.

Loading a single modern web page can require an unimaginable number of DNS query operations. For example, when we analyzed the ESPN homepage, we counted 93 distinct domain names, from a.espncdn.com to z.motads.com. All of them are necessary for the complete loading of the page!

To handle such a load on the query system that needs to serve the entire world, DNS is designed as a multi-level hierarchy. At the top of this pyramid are the root servers — each top-level domain, like .com, has its set of servers that are the ultimate authority for each domain below them. One level above these servers are the root servers themselves, from a.root-servers.net up to m.root-servers.net.

How often does this happen?

Due to the multi-level caching hierarchy of the DNS infrastructure, a very small percentage of global DNS queries reach the root servers. Most users obtain DNS resolver information directly from their provider. When a user's device needs to determine how to reach a specific website, the request is first sent to a DNS server managed by that local provider. If the local DNS server does not have an answer cached, it forwards the request to its own 'forwarding servers' (if configured).

If neither the local provider's DNS server nor the specified 'forwarding servers' have a cached response, the request is escalated directly to the authoritative server for the domain above that you are trying to resolve. In the case of domain.com this means that the request is sent to the authoritative servers for the domain itself com, located at gtld-servers.net.

System gtld-servers, the server being queried responds with a list of authoritative name servers for the domain domain.com, as well as at least one glue record containing the IP address of one such name server. The responses then cascade down the chain — each forwarding server passes these responses back down to the server that requested them, until the answer finally reaches the local provider's server and the user's computer. All of them cache this response to avoid unnecessarily bothering higher-level systems.

In most cases, name server records for domain.com will already be cached on one of these forwarding servers, so the root servers remain undisturbed. However, we are still talking about the familiar form of URL — the one that resolves to a standard website. Chrome requests relate to the level above of this, at the level of the clusters themselves root-servers.net.

Chromium and NXDomain hijacking checks

Chromium checks 'Is this DNS server misleading me?' make up almost half of all traffic reaching the Verisign root DNS server cluster.

The Chromium browser, the parent project of Google Chrome, the new Microsoft Edge, and countless lesser-known browsers, aims to provide users with a simplified search experience in a single input field, sometimes referred to as the 'Omnibox.' In other words, users can input both actual URLs and search queries in the same text field at the top of the browser window. Taking another step toward simplification, it also does not require users to enter part of the URL with http:// or https://.

As convenient as this may be, such an approach requires the browser to distinguish between what should be considered a URL and what is a search query. In most cases, this is quite obvious— for example, a string containing spaces cannot be a URL. However, it can be trickier when considering intranets—private networks that may also use private top-level domains to resolve actual websites.

If a user on their company intranet types 'marketing,' and there is an internal website with the same name within the company's intranet, Chromium displays a prompt asking whether the user wants to search for 'marketing' or go to https://marketing. This is somewhat acceptable, but many Internet service providers and public Wi-Fi providers 'hijack' each typosquatted URL entered, redirecting users to ad-laden pages.

Random Generation

Chromium developers did not want users on ordinary networks to see a prompt asking what they meant every time they searched for a single word, so they implemented a test: upon launching the browser or switching networks, Chromium performs DNS lookups for three randomly generated top-level 'domains' ranging from seven to fifteen characters long. If any two of these requests return with the same IP address, then Chromium assumes that the local network is 'hijacking' mistakes NXDOMAIN, which it should be receiving, so the browser considers all single-word queries as search attempts until further notice.

Unfortunately, in networks that do not hijack DNS query results, these three operations are typically elevated to the very top, all the way to the root name servers: the local server does not know how to resolve qwajuixk, which forwards this request to its forwarding server, which does the same thing until finally, a.root-servers.net either one of its 'brothers' has to say 'Sorry, but this is not a domain'.

Since there are approximately 1.67*10^21 possible fake domain names ranging from seven to fifteen characters, most often each of these tests conducted in a 'fair' network reaches the root server. This accounts for an impressive half of the total load on the root DNS, according to statistics from that part of the clusters root-servers.net, which belong to the company Verisign.

History repeats itself

This is not the first case where a project created with the best intentions overwhelmed or nearly overwhelmed a public resource with unnecessary traffic — it immediately reminded us of the long and sad history of D-Link and the Network Time Protocol (NTP) server of Poul-Henning Kamp in the mid-2000s.

In 2005, FreeBSD developer Poul-Henning, who also owned the only Stratum 1 Network Time Protocol server in Denmark, received an unexpected and hefty bill for the traffic transmitted. In short, the cause was that D-Link developers hardcoded the addresses of Stratum 1 NTP servers, including Kamp's server, into the firmware of their switches, routers, and access points. This instantly increased Kamp's server traffic ninefold, prompting the Danish Internet Exchange to change his rate from 'Free' to '9,000 dollars per year'.

The problem wasn't that there were too many D-Link routers, but that they disrupted the hierarchy. Just like DNS, NTP must operate in a hierarchical manner — Stratum 0 servers relay information to Stratum 1 servers, which in turn relay information to Stratum 2 servers, and so on down the hierarchy. A typical home router, switch, or access point like those that D-Link hardcoded the NTP server addresses into should have sent requests to Stratum 2 or Stratum 3 servers.

The Chromium project, probably with the best intentions, repeated the NTP problem in a DNS issue by overwhelming the root servers of the Internet with requests they were never meant to handle.

There is hope for a quick resolution

The Chromium project has an open bug, which requires disabling the Intranet Redirect Detector by default to resolve this issue. Credit goes to the Chromium project: the bug was discovered before, as Matt Thomas from Verisign brought significant attention to it with his post post on the APNIC blog. The bug was reported in June but faded into obscurity until Thomas's post; after it, the issue began to receive close scrutiny.

There is hope that the problem will be resolved soon, and root DNS servers will no longer have to handle around 60 billion fake requests daily.