Cloudflare, provides content delivery network services for 27 million internet resources and handles 13% of traffic from the top 1000 websites, detailed the incident during which many segments of the Cloudflare network were disrupted for 27 minutes, including those responsible for delivering traffic to London, Chicago, Los Angeles, Washington, Amsterdam, Paris, Moscow, and St. Petersburg. The problem was caused by an incorrect configuration change on a router in Atlanta. During the incident, which occurred on July 17 from 21:12 to 21:39 (UTC), the overall traffic volume on the Cloudflare network decreased by approximately 50%.
During technical work, seeking to offload some traffic from one of the backbones, engineers deleted a line in the configuration block that defined the list of routes accepted through the backbone, filtered according to a specified prefix list. The correct action would have been to deactivate the entire block, but mistakenly only the line with the prefix list was removed.
{master}[edit]
atl01# show | compare
[edit policy-options policy-statement 6-BBONE-OUT term 6-SITE-LOCAL from]
! inactive: prefix-list 6-SITE-LOCAL { … }
Block content:
from {
prefix-list 6-SITE-LOCAL;
}
then {
local-preference 200;
community add SITE-LOCAL-ROUTE;
community add ATL01;
community add NORTH-AMERICA;
accept;
}
Due to the removal of the binding to the prefix list, the remaining part of the block began to propagate to all prefixes, and the router started sending all its BGP routes to the routers of other backbones. Coincidentally, the new routes had a higher priority (local-preference 200) compared to the priority (100) set for other routes by the automated traffic optimization system. As a result, instead of removing routing from the backbone, there was a leak of higher-priority BGP routes, causing traffic directed to other backbones to go to Atlanta, leading to router overload and a partial network collapse.
To prevent similar incidents in the future, several changes are set to be made to the Cloudflare backbone settings on Monday. A limit on the maximum number of prefixes (maximum-prefix) will be added for BGP sessions, which will block the problematic backbone in case it routes too many prefixes. If this limit had been implemented earlier, the issue in question would have caused the backbone in Atlanta to go down, but it would not have affected the overall functioning of the network, as the Cloudflare network is designed to handle the failure of individual backbones. Among the changes already implemented, there is a revision of the local preference for local routes, which will prevent one router from influencing traffic in other parts of the network.
Source: opennet.ru
