Constantin Riabțev, administrator of the infrastructure at kernel.org, published the traffic analysis results for the service git.kernel.org, which allows viewing the contents of git repositories related to the development of the Linux kernel. The service processes about 6 million requests for commit information daily, of which 66% come from bots that can be blocked using the Anubis system. 33% of the requests pass through the Anubis system, allowing entry only after solving a JavaScript challenge where the SHA-256 hash in combination with a server-issued string contains a specific number of leading zeros (this task requires CPU resources to solve, but not for verification).
It is impossible to precisely determine who among these 33% are humans and who are advanced bots, but based on the nature of the requests, it is concluded that the overall share of legitimate requests is about 2%, while the remaining 98% come from scrapers. One sign of bots is direct requests for old commits in random old branches, which people are unlikely to need in their work. Bot activity creates an immense parasitic load on servere since instead of downloading all the code and commit history once through the 'git clone' operation, bots send billions of requests, trying all options through the web interface and repeatedly requesting the same data with different parameters.
Blocking bots via adrese IP. and autonomous systems quickly stopped working, as bots switched to sending requests from millions of random IP addresses from home or mobile networks, obtained through botnet activity or monetization of browser extensions. Only 4-5 requests are sent from each address, making it futile to block them at the firewall level.
For a while, the Anubis system helped in blocking bots, but over time, bots adapted to perform the computations proposed by Anubis. Among other things, bots learned to solve the level 5 difficulty task, and it is problematic to increase the level further, as even the fifth level requires several seconds of computation and significantly irritates legitimate users.
În concluzie, întreținerea cererilor scraper-ilor în infrastructura kernel.org necesită mai multe resurse decât toate celelalte tipuri de operațiuni legitime, inclusiv "git clone". Pe 5 servere, serverele care deservesc git.kernel.org, în orice moment, 14-16 din cele 90 de nuclee CPU sunt constant ocupate cu afișarea commit-urilor git pentru boți. Pentru a reduce povara, administratorii încearcă acum să diminueze capacitățile serviciului, să dezactiveze unele operațiuni consumatoare de resurse, să limiteze accesul anonim și să reducă numărul de linkuri disponibile pentru navigare.
Sursa: opennet.ro
