Google has made it possible to prohibit the crawling of a website by robots used for training the company's neural networks.
You can hide the content of the site from robots. Bard and VertexAI, and this prohibition will not affect the site's indexing by the search engine itself. To do this, you need to add the corresponding entry to the robots.txt.
With the expansion of its AI model base, Google plans to automatically extend the ability to prohibit indexing of a site by these AIs.
Source: linux.org.ru
