Konstantin Ryabtsev, infrastructure administrator at kernel.org, published the results of analyzing the traffic of the git.kernel.org service, which allows viewing the contents of git repositories related to Linux kernel development. The service processes about 6 million requests for commit information daily, of which 66% come from bots, which can be blocked using the Anubis system. 33% of requests pass through the Anubis system, which allows access only after finding a value in JavaScript whose SHA-256 hash, combined with a string issued by the server, contains a certain number of leading zeros (this task requires CPU resources for solving but not for verification).
It is impossible to accurately determine who among these 33% are humans and who are advanced bots, but based on the nature of the requests, it is concluded that the overall share of legitimate requests is about 2%, while the remaining 98% are from scrapers. One of the signs of bots is direct requests for old commits in random old branches, which are unlikely to be needed by humans in their work. Bot activity creates a huge parasitic load on servers because instead of downloading all the code and commit history once through the 'git clone' operation, bots send billions of requests, trying all variations of access through the web interface and repeatedly requesting the same data with different parameters.
Blocking bots by IP addresses and autonomous systems quickly stopped working, as bots switched to sending requests from millions of random IP addresses from home or mobile networks, obtained through botnet activities or monetization of browser extensions. Only 4-5 requests are sent from each address, making it useless to block them at the firewall level.
For a while, the Anubis system helped block bots, but over time, bots adapted to perform the computations proposed in Anubis. Among other things, bots learned to solve tasks at level 5 difficulty, and further increasing the level is problematic, as even the fifth level requires several seconds of calculations and greatly irritates legitimate users.
Ultimately, servicing requests from scrapers on the kernel.org infrastructure consumes more resources than all other legitimate operations, including 'git clone'. On 5 servers, serving git.kernel.org, at any given time, 14-16 out of the 90 available CPU cores are constantly occupied with rendering git commits for bots. To reduce the load, administrators are now trying to limit the capabilities of the service, disable some resource-intensive operations, restrict anonymous access, and reduce the number of navigable links.
Source: opennet.ru
