Konstantin Ryabtsev, administrator of the infrastructure at kernel.org, published the results of an analysis of the traffic for the git.kernel.org service, which allows viewing the contents of git repositories related to the development of the Linux kernel. The service handles about 6 million request information about commits daily, of which 66% come from bots, which can be blocked using the Anubis system. 33% of requests undergo a check by the Anubis system, which only allows entry after a JavaScript value is correctly guessed, the SHA-256 hash of which, combined with the string provided by the server, contains a specific number of leading zeros (this task requires CPU resources to solve but does not require resources for verification).
It's impossible to accurately determine who among these 33% are humans and who are advanced bots, but based on the nature of the requests, it is concluded that the overall share of legitimate requests is about 2%, while the other 98% come from scrapers. One of the signs of bots is direct requests for old commits in random old branches, which are unlikely to be needed by people in their work. The activity of bots creates a huge parasitic load on serverët because instead of downloading all the code and commit history once via the 'git clone' operation, bots send billions of requests, iterating through all variants via the web interface and repeatedly requesting the same data with different parameters.
Blocking bots by IP adresat and autonomous systems quickly stopped working, as bots switched to sending requests from millions of random IP addresses from home or mobile networks, obtained through botnet activity or monetization of browser extensions. From each address, only 4-5 requests are sent, so blocking them at the firewall level is useless.
For a while, the Anubis system helped to block bots, but over time, the bots adapted to perform the computations suggested in Anubis. Among other things, bots learned to solve the 5th level difficulty task, and further increasing the level is problematic, as even the fifth level requires several seconds of computation and greatly irritates legitimate users.
Në fund, shpenzimet për shërbimin e kërkesave të scrapperëve në infrastrukturën e kernel.org janë më të larta se të gjitha llojet e tjera të operacioneve legjitime, përfshirë "git clone". Në 5 sistemet, shërbimi git.kernel.org ka nga 14-16 nga 90 bërthamat e CPU gjithmonë të angazhuar në shfaqjen e git-komiteteve për botët. Për të ulur ngarkesën, administratoret tani përpiqen të kufizojnë mundësitë e shërbimit, të çaktivizojnë disa operacione që konsumojnë shumë burime, të kufizojnë aksesin anonim dhe të reduktojnë numrin e lidhjeve të disponueshme për navigimin.
Burimi: opennet.ru
