Recent times have seen a significant increase in the activity of web bots that index traffic. In addition to properly functioning bots, "ferocious" bots have emerged, disregarding the indexing rules set by robots.txt, operating from tens of thousands of different IPs, posing as legitimate users, and not adhering to a reasonable request intensity policy. These bots create a huge parasitic load on servers, disrupt the normal operation of systems, and consume the time of administrators. The activity of such bots is perceived by many as malicious behavior.
As a measure to slow down the operation of such bots, as well as bots scanning unpatched vulnerabilities in typical web applications, one of the administrators proposed the method of "zip bombs." The essence of the method is that in response to a web bot's request for a page, content is transmitted that is effectively compressed using the "deflate" method, the size of which upon unpacking greatly exceeds the size of the data transmitted over the network. For instance, using the "deflate" method, content from /dev/zero packed into 10 MB will require 1 GB of disk space upon unpacking. With the use of the "brotli" compression method, a level has been achieved where transmitting 81 MB results in unpacking 100 TB of data.
This type of protection can be activated by creating traps that are accessible via invisible links marked with the flag 'rel="nofollow"', excluded from indexing by robots.txt, and triggered at a sufficiently high recursion level for bots attempting to pose as ordinary users. In practice, the proposed method is not recommended, as the site may be blacklisted by Google and marked as harmful in the Chrome browser with the 'Safe Browsing' mode enabled.
Sursa: opennet.ro
