Cloudflare Company report on yesterday's incident, which resulted in From 13:34 to 16:26 (MSK), there were problems accessing many resources on the global network, including the infrastructure of Cloudflare, Facebook, Akamai, Apple, Linode, and Amazon AWS. Problems in the Cloudflare infrastructure, which provides a CDN for 16 million sites, from 14:02 to 16:02 (MSK). Cloudflare estimated that approximately 15% of global traffic was lost during the outage.
The problem was route leak through BGP, during which about 20 thousand prefixes for 2400 networks were incorrectly redirected. The source of the leak was the provider DQE Communications, which used software to optimize routing. BGP Optimizer breaks IP prefixes into smaller ones, for example, splits 104.20.0.0/20 into 104.20.0.0/21 and 104.20.8.0/21, and as a result, DQE Communications kept a large number of specific routes on its side that override more common routes (i.e. instead of generic routes to Cloudflare, more granular routes to specific Cloudflare subnets were used).
These point routes were advertised to one of the customers (Allegheny Technologies, AS396531) who also had a connection through another provider. Allegheny Technologies broadcast the received routes to another transit provider (Verizon, AS701). Due to the lack of proper filtering of BGP announcements and the limit on the number of prefixes, Verizon picked up this announcement and broadcast the received 20 thousand prefixes to the rest of the Internet. Incorrect prefixes, due to their granularity, were perceived as higher priority, since a specific route has a higher priority than a general one.
As a result, traffic for many large networks began to be directed through Verizon to a small provider DQE Communications, unable to handle the flooding traffic, which led to a collapse (the effect is comparable to replacing part of a busy freeway with a country road).
To prevent similar incidents from occurring in the future
:
- Use announcements based on RPKI (BGP Origin Validation, allows receiving announcements only from network owners);
- Limit the maximum number of accepted prefixes for all EBGP sessions (setting maximum-prefix would help to immediately discard the transmission of 20 thousand prefixes within one session);
- Apply filtering based on the IRR registry (Internet Routing Registry, determines AS through which routing of given prefixes is allowed);
- Use default deny settings ('default deny') recommended in RFC 8212 on routers;
- Stop reckless use of BGP optimizers.
Source: opennet.ru