Konstantin Ryabtsev, administrator of the kernel.org infrastructure, published the traffic analysis results of the git.kernel.org service, which allows viewing the contents of git repositories related to kernel development. The service processes about 6 million requests for commit information daily, of which 66% are from bots that can be blocked by the Anubis system. 33% of requests undergo verification by the Anubis system, which only permits access after solving a JavaScript-based problem, where the SHA-256 hash combined with a string provided by the server contains a specified number of leading zeros (this task requires CPU resources for solution but does not require resources for verification).
It is impossible to precisely determine who among these 33% are humans and who are advanced bots, but based on the nature of the requests, it is concluded that the overall share of legitimate requests is approximately 2%, while the remaining 98% come from scrapers. One of the signs of bots is direct requests for old commits in random outdated branches, which are unlikely to be needed by people in their work. The activity of bots creates a massive parasitic load on servere because instead of loading all the code and commit history once through the 'git clone' operation, bots send billions of requests, iterating through all possible calls via the web interface and repeatedly requesting the same data with different parameters.
Bot blocking through IP-naslovov and autonomous systems quickly became ineffective, as bots switched to sending requests from millions of random IP addresses from home or mobile networks acquired through botnet activity or browser extension monetization. From each address, only 4-5 requests are sent, making it useless to block them at the firewall level.
For some time, the Anubis system helped block bots, but over time, bots adapted to perform the computations required in Anubis. Among other things, bots learned to solve the task of level 5 complexity, and it becomes challenging to increase the level further, as even the fifth level requires several seconds of computation and significantly annoys legitimate users.
Kot na razvoj infrastrukture kernel.org porabimo več sredstev za upravljanje zahtev botov kot za vse druge vrste legitimnih operacij, vključno z 'git clone'. Na 5 serverima, ki upravljajo git.kernel.org, je v vsakem trenutku 14-16 od 90 jedrnih procesorjev trajno zasedenih z izpisovanjem git-commitov za bote. Da bi zmanjšali obremenitev, administratorji zdaj skušajo omejiti možnosti storitve, izklopiti nekatere vire, omejiti anonimni dostop ter skrajšati število povezav, dostopnih za navigacijo.
Izvor: opennet.ru
