Konstantin Ryabtsev, the administrator of the kernel.org infrastructure, published the results of an analysis of the traffic of the git.kernel.org service, which allows viewing the contents of Git repositories of projects related to kernel development. LinuxThe service processes approximately 6 million commit information requests daily, 66% of which are from bots, which are blocked by the Anubis system. 33% of requests are verified by the Anubis system, which allows entry only after brute-forcing a JavaScript value whose SHA-256 hash, combined with the server-sent string, contains a certain number of leading zeros (this task requires CPU resources to solve, but does not require resources for verification).
It's impossible to determine exactly which of these 33% are humans and which are advanced bots, but based on the nature of the requests, it's been concluded that the overall share of legitimate requests is approximately 2%, with the remaining 98% coming from scrapers. One indicator of bots is direct requests for old commits in random old branches that are unlikely to be needed by humans. Bot activity creates a huge parasitic load on servers Because instead of downloading all the code and commit history once via the "git clone" operation, bots send billions of requests, trying all the possible options through the web interface and repeatedly requesting the same data with different parameters.
Blocking bots by IP addresses and autonomous systems quickly stopped working, as bots switched to sending requests from millions of random IP addresses from home or mobile networks, obtained through botnet activity or monetization of browser add-ons. Each address sends only 4-5 requests, making blocking them at the firewall level useless.
For a while, the Anubis system helped block bots, but over time, bots adapted to perform the calculations offered by Anubis. Among other things, bots learned to solve problems of level 5 difficulty, but increasing the difficulty level is problematic, as even level 5 requires several seconds of calculation and is highly annoying for legitimate users.
As a result, servicing scraper requests in the kernel.org infrastructure consumes more resources than all other types of legitimate operations, including "git clone." серверахAt any given time, 14-16 of the 90 CPU cores serving git.kernel.org are constantly busy displaying Git commits to bots. To reduce the load, administrators are now trying to scale back the service's capabilities, disable some resource-intensive operations, restrict anonymous access, and reduce the number of links available for navigation.
Source: opennet.ru
