Billede:
For aggregator websites in e-commerce, it is crucial to maintain up-to-date information. Otherwise, the main advantage they have – the ability to see the most relevant data in one place – will disappear.
To solve this issue, it is necessary to use web scraping techniques. The idea is that special software – a crawler – is created to visit the necessary sites from a list, parse information from them, and upload it to the aggregator website.
The problem is that often the site owners from whom aggregators take data do not want to make accessing their information easily. This is understandable – if the price information in an online store appears on an aggregator site and is higher than in competing stores shown there, the business will lose customers.
Methods of counteracting scraping
Therefore, often the owners of such sites counteract scraping – that is, the downloading of their data. They can identify requests sent by bot crawlers by IP address. Usually, such software uses so-called server IPs, which are easy to detect and block.
In addition, instead of blocking requests, another method is often used – irrelevant information is shown to detected bots. For example, prices for products can be artificially inflated or deflated, or their descriptions changed.
An example frequently cited in this context is airline ticket prices. Indeed, airlines and travel agencies can often show different results for the same flights depending on the IP address. A real case: searching for a flight from Miami to London on the same date from an IP address in Eastern Europe and Asia returns different results.
In the case of an IP address in Eastern Europe, the price looks like this:
And for an IP address from Asia, it looks like this:
As can be seen, the price for the same flight differs significantly – the difference is $76, which is quite substantial. For an aggregator website, nothing is worse than this – if incorrect information is presented on it, users will not use it. Moreover, if a specific product has one price on the aggregator, but it changes when redirected to the seller's site – this also negatively impacts the project's reputation.
Solution: using residential proxies
For data scraping needs, issues can be avoided by using residential proxies. Server IPs are provided by hosting providers. Identifying whether an address belongs to a specific provider's pool is quite simple – each IP has an ASN number that contains this information.
There are many services for analyzing ASN numbers. Often, they are integrated with anti-bot systems that block access to crawlers or manipulate the data returned in response to their requests.
Residential IP addresses help bypass such systems. These IPs are issued by internet providers to homeowners, with the appropriate markings in all related databases. There are specialized residential proxy services that allow you to use residential addresses. – just such a service.
Requests that aggregators' crawlers send from residential IPs appear as if they are coming from regular users in a specific region. And regular visitors are not blocked – in the case of online stores, they are potential customers.
As a result, using rotating proxies from Infatica allows aggregator sites to receive guaranteed accurate data and avoid blocks and parsing difficulties.
Other articles on the topic of using residential proxies for business:
Kilde: habr.com
