The internet seems like a strong, independent, and indestructible structure. In theory, the network's strength is enough to survive a nuclear explosion. In reality, the internet can be brought down by a single small router. This is due to the fact that the internet is a jumble of contradictions, vulnerabilities, errors, and cat videos. The backbone of the internet, the BGP protocol, has a host of issues. It's surprising that it still functions. In addition to errors within the internet itself, it is also attacked by anyone who can: major internet providers, corporations, governments, and DDoS attacks. What can we do about this and how do we cope with it?
The answer lies with Alexey Uchakov () â the leader of the network engineering team at IQ Option. His main task is to ensure platform availability for users. In Alexey's presentation at , we will talk about BGP, DDoS attacks, the internet kill switch, provider errors, decentralization, and instances where a small router sent the internet to sleep. In the end, there will be a few tips on how to survive all this.
The Day the Internet Broke
I will present just a few incidents when internet connectivity broke down. This will be enough for a complete picture.
"The AS7007 Incident". The internet first broke in April 1997. There was a bug in the software of a router from autonomous system 7007. At some point, the router announced its internal routing table to its neighbors and sent half of the network into a black hole.
"Pakistan vs. YouTube". In 2008, brave folks from Pakistan decided to block YouTube on their end. They did it so well that half the world was left without cat videos.
"The Prefix Hijacking of VISA, MasterCard, and Symantec by Ros Telecom". In 2017, Ros Telecom mistakenly began announcing the prefixes for VISA, MasterCard, and Symantec. As a result, financial traffic was routed through channels controlled by the provider. The leak didn't last long, but it was unpleasant for the financial companies.
"Google vs. Japan". In August 2017, Google began announcing the prefixes of major Japanese providers NTT and KDDI for part of its uplinks. The traffic was sent to Google as transit, likely by mistake. Since Google is not a provider and does not pass transit traffic, a significant portion of Japan was left without internet.
"DV LINK hijacked the prefixes of Google, Apple, Facebook, Microsoft". In 2017, the Russian provider DV LINK oddly began advertising the networks of Google, Apple, Facebook, Microsoft, and some other major players.
"eNet from the USA captured the prefixes of AWS Route53 and MyEtherwallet". In 2018, a provider from Ohio or one of its clients announced the networks of Amazon Route53 and the cryptocurrency wallet MyEtherwallet. The attack was successful: despite the self-signed certificate, which warned users when accessing the MyEtherwallet site, many wallets were hijacked and some cryptocurrency was stolen.
There were more than 14,000 such incidents in 2017 alone! The network is still decentralized, so not everything and not everyone is affected. However, incidents occur by the thousands, and all of them are related to the BGP protocol, on which the internet operates.
BGP and its problems
Protocol BGP - Border Gateway Protocol, was first described in 1989 by two engineers from IBM and Cisco Systems on three "napkins" - A4-sized sheets of paper. These still lie in the main office of Cisco Systems in San Francisco as a relic of the networking world.
At the core of the protocol is the interaction of autonomous systems â Autonomous Systems or simply AS. An autonomous system is just an ID that is linked to IP networks in the public registry. A router with this ID can announce these networks to the world. Accordingly, any route on the internet can be represented as a vector, known as AS Path. The vector consists of the numbers of autonomous systems that must be traversed to reach the destination network.
For example, there is a network consisting of a certain number of autonomous systems. We need to get from system AS65001 to system AS65003. The path from one system is represented as AS Path in the diagram. It consists of two autonomous systems: 65002 and 65003. For each destination address, there is an AS Path vector consisting of the numbers of the autonomous systems we need to pass through.
So what are BGP's problems?
BGP is a trust-based protocol
The BGP protocol is trust-based. This means that we inherently trust our neighbor. This is a feature of many protocols developed in the early days of the internet. Let's understand what "trust" means.
There is no authentication of neighbors. Formally, there is MD5, but MD5 in 2019 is quite questionable...
There is no filtering. BGP has filters and they are documented, but they are not used or are used incorrectly. I'll explain later why.
It's very easy to set up peering.. Configuring peering in BGP on almost any router is just a couple of lines of config.
No privileges required to manage BGP.. You don't have to take exams that confirm your qualifications. No one will revoke your rights for configuring BGP while drunk.
Two main problems.
Prefix hijacks.. A prefix hijack is the announcement of a network you do not own, as in the case of MyEtherwallet. We took some prefixes, negotiated with the provider or hacked it, and through it, we announce these networks.
Route leaks.. Route leaks are a bit more complicated. A leak is a change in the AS Path.. In the best case, the change will lead to greater latency because it has to traverse a longer route or a less efficient link. In the worst case, it could repeat the situation with Google and Japan.
Google itself is not an operator and not a transit autonomous system. But when it announced to its provider the networks of Japanese operators, the traffic through Google was seen as more priority due to the AS Path. The traffic went there and was dropped simply because the routing settings within Google are more complex than just filters at the edge.
Why do filters not work?
Nobody cares.. This is the main reasonâeveryone just doesn't care. An admin from a small provider or a company that connected to a provider via BGP took MikroTik, configured BGP on it, and doesn't even know that filters can be configured there.
Configuration errors.. Something was debugged, there was a mistake in the mask, the wrong network was setâand here's another error.
No technical capacity.. For instance, communication providers have many clients. It would make sense to automatically update filters for each clientâkeeping track of when they acquire a new network, that they have leased their network to someone. Monitoring this is difficult, and doing it manually is even harder. That's why they simply set relaxed filters or don't set any filters at all.
Exceptions. There may be exceptions for favored and large clients. Especially in the case of inter-operator interfaces. For example, Transtelecom and Rostelecom have a ton of networks, and thereâs an interface between them. If the interface fails, it wonât be good for anyone, so they relax or completely remove the filters.
Outdated or irrelevant information in IRR.. Filters are built based on the information that is recorded in. IRR - Internet Routing Registry. These are registries of regional internet registrars. Often, the registries contain outdated or irrelevant information, or sometimes both.
Who are these registrars?
All internet addresses belong to organizations IANA - Internet Assigned Numbers Authority. When you buy an IP network from someone, you are not purchasing the addresses, but the right to use them. Addresses are an intangible resource, and by common agreement, they are all owned by the IANA agency.
The system works like this. IANA delegates the management of IP addresses and autonomous system numbers to five regional registrars. Those registrars issue autonomous systems LIR - Local Internet Registrars. Then, LIR allocates IP addresses to end users.
The drawback of the system is that each regional registrar maintains its registries in its own way. Everyone has their views on what information should be included in the registries, who should or should not verify it. As a result, we have the chaos that exists now.
How else can we tackle these issues?
IRR - mediocre quality. It's clear with IRR - everything is bad there.
BGP communities. This is a certain attribute described in the protocol. We can attach, for example, a special community to our announcement so that neighbors do not send our networks to their neighbors. When we have a P2P link, we exchange only our networks. To prevent a route from accidentally going to other networks, we attach a community.
Communities are not transitive. It's always a deal between two parties, and that's their drawback. We cannot attach any community except for one that is accepted by default by everyone. We cannot be sure that this community will be accepted and interpreted correctly. Therefore, at best, if you negotiate with your uplink, he will understand what you want from him regarding the community. But his neighbor might not understand it, or the operator might simply drop your tag, and you won't achieve what you wanted.
RPKI + ROA only solves a small part of the problems. RPKI is Resource Public Key Infrastructure â a special framework for signing routing information. It's a good idea to make LIRs and their clients maintain an up-to-date database of address space. But there is one problem with it.
RPKI is also a hierarchical system of public keys. IANA has the key from which RIR keys are generated, and from them, LIR keys are created. They use these to sign their address space with ROAs â Route Origin Authorizations:
â I assure you that this prefix will be announced on behalf of this AS.
Besides ROA, there are other objects, but we can discuss them later. It seems that the concept is good and useful. However, it does not protect us from leaks at all and does not solve all issues with prefix hijacking. Therefore, players are not in a hurry to implement it. Although there are already assurances from major players like AT&T and large IXes that prefixes with an invalid ROA record will be dropped.
They may do this, but for now we have a huge number of prefixes that are not signed at all. On one hand, it's unclear whether they are being announced validly. On the other hand, we cannot drop them by default because we are unsure whether it's right or not.
What else is there?
BGPSec. This is a cool concept created by academics for the network of pink ponies. They said:
â We have RPKI + ROA â a mechanism for signing address space. Letâs create a separate BGP attribute and call it BGPSec Path. Each router will sign its announcements with its own signature before announcing them to neighbors. This way, we will get a trusted path from the chain of signed announcements and can verify it.
In theory, it sounds good, but in practice, there are many problems. BGPSec disrupts many existing BGP mechanisms for selecting the next hop and managing incoming/outgoing traffic directly on the router. BGPSec does not work until 95% of all market participants implement it, which is, by itself, a utopia.
BGPSec has huge performance issues. With current hardware, the speed of verifying announcements is about 50 prefixes per second. For comparison: the current internet table with 700,000 prefixes would take 5 hours to load, during which time 10 more updates would occur.
BGP Open Policy (Role-based BGP). A fresh proposal based on the Gao-Rehkford. These are two researchers studying BGP.
The Gao-Rehkford model states the following. To simplify, in the case of BGP, there are a small number of interaction types:
- Provider Customer;
- P2P;
- internal interaction, let's say, iBGP.
Based on the role of the router, some import/export policies can already be applied by default. The administrator does not need to configure prefix lists. Based on the role agreed upon between the routers and the one that can be set, we already get some default filters. Currently, this is a draft being discussed in the IETF. I hope we will soon see it as an RFC and implemented in hardware.
Major Internet Service Providers
Let's consider the provider CenturyLink. This is the third-largest provider in the U.S., servicing 37 states and operating 15 data centers.
In December 2018, CenturyLink was down on the U.S. market for 50 hours. During the incident, there were issues with ATM operations in two states, and the 911 service was unavailable for several hours in five states. Additionally, a lottery in Idaho was disrupted due to this incident. The U.S. FCC is currently investigating this matter.
The cause of the tragedy was a single network card in one data center. The card failed, sending incorrect packets, and all 15 of the provider's data centers went down.
The idea of this provider being too big to faildid not work at all. You can take any large player and take them down with a small issue. In the U.S., the connectivity situation is still good. CenturyLink customers who had backups massively switched over to them. Later, alternative operators complained about the overload of their links.
If a hypothetical 'Kazakhtelecom' goes down, the entire country will lose internet access.
Corporations
Does the internet rely on Google, Amazon, Facebook, and other corporations? No, they also break it.
In 2017, at the ENOG13 conference in St. Petersburg, Jeff Houston from APNIC presented In it, he stated that we are accustomed to thinking that interactions, flows of money, and traffic on the internet are vertical. We have small providers paying for connectivity to larger ones, and those paying for connectivity to global transit.
Currently, we have such a vertically-oriented structure. All would be well, but the world is changing â large players are building their own transoceanic cables to construct their own backbones.
News about the CDN cable.
In 2018, TeleGeography released a study indicating that more than half of the internet traffic is no longer internet but backbones of large CDN players. This traffic is related to the internet, but it is no longer the network we used to talk about.
The internet is fragmenting into a large set of loosely connected networks.
Microsoft has its own network, Google has its own, and they barely intersect with each other. Traffic that originates somewhere in the USA travels through Microsoft's channels across the ocean to Europe, somewhere at a CDN, then it connects through the CDN or IX with your provider and reaches your router.
Decentralization is disappearing.
This strong aspect of the internet, which could help it survive a nuclear explosion, is being lost. Places of concentration of users and traffic are emerging. If a major player like Google Cloud goes down, many will suffer. We partially felt this when Roskomnadzor blocked AWS. The case of CenturyLink shows that even a small issue can be enough.
Previously, not everything broke and not for everyone. In the future, we may reach a point where affecting one major player could significantly disrupt many services for many users.
States
Next in line are states, and this usually happens like this.
Here, our Roskomnadzor is not even a pioneer. Such practices as Internet shutdown exist in Iran, India, and Pakistan. In England, there is a bill regarding the ability to disconnect the internet.
Any large state desires to have a switch to disconnect the internet either completely or in parts: Twitter, Telegram, Facebook. It's not so much that they don't understand they will never achieve this, but they really want to. The switch is usually applied for political purposes â to eliminate political competitors, or during elections, or because Russian hackers broke something again.
DDoS attacks
I wonât take away the bread from my colleagues at Qrator Labs; they do this much better than I do. They have on internet stability. Hereâs what they wrote in the report for 2018.
The average duration of DDoS attacks has decreased to 2.5 hours. Attackers are also starting to count their money, and if the resource doesnât go down immediately, they quickly leave it alone.
The intensity of attacks is increasing. In 2018, we saw 1.7 Tb/s on the Akamai network, and this is not the limit.
New attack vectors are emerging and old ones are strengthening.New amplification-prone protocols are appearing, and new attacks on existing protocols are emerging, especially against TLS and similar ones.
The majority of traffic comes from mobile devices.At the same time, internet traffic is shifting to mobile clients. Both attackers and defenders need to adapt to this.
There are no invulnerable systems.This is the main point â there is no universal protection that will guarantee safety against any DDoS.
A system cannot be taken down as long as it is not connected to the internet.
I hope I have sufficiently frightened you. Now, let's think about what to do about it.
What to do?!
If you have some free time, a desire, and knowledge of English â participate in working groups: IETF, RIPE WG. These are open mailing lists; subscribe to their newsletters, join discussions, attend conferences. If you have LIR status, you can vote, for example, in RIPE for various initiatives.
For ordinary people â this is. monitoringto know what broke.
Monitoring: what to check?
Regular Pingnot only for binary checks â to see if it works or not. Record RTT in history to identify anomalies later.
Tracerouteis a utility for determining the routes of data packets in TCP/IP networks. It helps to identify anomalies and blocks.
HTTP checks for custom URLs and TLS certificates will help reveal blocks or DNS spoofing for attacks, which are practically the same. Blocks are often implemented by DNS spoofing and redirecting traffic to a placeholder page.
If possible, check the DNS resolution of your origin from different locations if you have an application. This way, you will discover interception anomalies in DNS, which providers sometimes introduce.
Monitoring: where to check from?
There is no universal answer. Check from where your users are coming. If users are in Russia â check from Russia, but do not limit yourself to it. If your users live in different regions â check from those regions. But it's better to check from all over the world.
Monitoring: what to check with?
I have come up with three methods. If you know more, please write in the comments.
- RIPE Atlas.
- Commercial monitoring.
- Your own network of virtual machines.
Let's talk about each of them.
RIPE Atlas â it's a small box. For those familiar with the domestic 'Inspector', it's the same kind of box but with a different sticker.
RIPE Atlas â a free program. You register, receive a router by mail, and connect it to your network. For allowing someone else to use your probe, you receive some credits. With these credits, you can conduct your own research. You can test in various ways: ping, traceroute, checking certificates. The coverage is quite vast, with many nodes. However, there are nuances.
The credit system does not allow for production solutions. Credits are insufficient for ongoing research or commercial monitoring. They are enough for short research or a one-time check. The daily limit from one probe is consumed by 1-2 checks.
Coverage is uneven. Since the program is free in both directions, the coverage is good in Europe, the European part of Russia, and some regions. But if you need Indonesia or New Zealand, it's significantly worse â 50 probes per country may not be enough.
You cannot check HTTP from a probe. This is due to technical nuances. They promise to fix it in a new version, but for now, HTTP checks are not possible. You can only check the certificate. Some HTTP checks can only be made through a special RIPE Atlas device called Anchor.
The second way is commercial monitoring. It's good, since you're paying money, right? They promise you several dozen or hundreds of monitoring points around the world, creating nice dashboards out of the box. But again, there are problems.
It's paid, and in some places, very. Ping monitoring, checks from all over the world, and numerous HTTP checks can cost several thousand dollars a year. If your finances allow and you like this solution â go ahead.
Coverage may be insufficient in the area of interest. For ping checks, they can only specify the maximum abstract area â Asia, Europe, North America. Rare monitoring systems may detail the probe down to a specific country or region.
Weak support for custom tests. If you need something custom, rather than just a simple 'ping' on a URL, there are issues with this as well.
The third way is your own monitoring. It's classic: 'Why don't we write our own!'
Your monitoring turns into software product development, and itâs distributed. You're looking for an infrastructure provider, figuring out how to deploy and monitor itâmonitoring must be monitored, right? And support is also required. Think twice before you embark on this. It might be easier to pay someone to do it for you.
Monitoring BGP anomalies and DDoS attacks
Here, regarding available resources, itâs even simpler. BGP anomalies are detected using specialized services like QRadar and BGPmon.They receive full view tables from multiple operators. Based on what they see from different operators, they can detect anomalies, search for amplifiers, and more. Usually, registration is freeâyou enter your AS number, sign up for email notifications, and the service alerts you about your problems.
In monitoring DDoS attacks, it's also simple. Usually, it's NetFlow-based and logs.There are specialized systems like FastNetMon, modules for Splunk. As a last resort, thereâs your DDoS protection provider. You can also stream NetFlow to them, and based on that, they will notify you of attacks directed at you.
Conclusions
Don't harbor illusionsâ the internet will inevitably break down.Not everything will fail and not everyone will experience it, but 14,000 incidents in 2017 suggest that incidents will occur.
Your task is to notice problems as early as possible.At a minimum, not later than your user. Not only is it essential to notice, but always have a âPlan Bâ ready. The plan is a strategy of what you will do when everything fails:backup operators, data centers, CDNs. The plan is a separate checklist through which you check the operation of everything. The plan should work without involving network engineers, because they are usually few, and they want to sleep.
Thatâs all for now. I wish you high availability and green monitoring.
Next week in Novosibirsk, sun is expected, with high load and a high concentration of developers at In Siberia, a wave of talks about monitoring, availability, testing, security, and management is forecasted. Expect precipitation in the form of written notes, networking, photos, and social media posts. We recommend postponing all matters on June 24 and 25 and We look forward to seeing you in Siberia!
Source: habr.com
