Image:
For aggregator sites in the e-commerce sector, it is crucial to maintain up-to-date information. Otherwise, their main advantage – the ability to see the most relevant data in one place – disappears.
To address this issue, it is necessary to use web scraping techniques. The idea is to create special software – a crawler – that navigates through a list of required websites, parses information from them, and uploads it to the aggregator site.
The problem is that often the website owners from whom aggregators collect data do not want to provide access so easily. This is understandable – if pricing information from an online store appears on an aggregator site and is higher than that of the competitors also listed there, the business will lose customers.
Methods to combat scraping
Therefore, website owners often combat scraping – that is, the downloading of their data. They can identify requests sent by bot crawlers by IP address. Typically, such software uses so-called server IPs, which are easy to detect and block.
Moreover, instead of blocking requests, another method is often employed – irrelevant information is shown to identified bots. For example, they may inflate or deflate product prices or alter their descriptions.
A common example cited in this context is airline ticket prices. Indeed, airlines and travel agencies often display different results for the same flights depending on the IP address. A real case: searching for a flight from Miami to London on the same date using an IP address from Eastern Europe returns different results than an IP address from Asia.
For the IP address from Eastern Europe, the price looks like this:
And for the IP address from Asia, it looks like this:
As seen, the price for the same flight differs significantly – the difference is $76, which is quite substantial. For an aggregator site, nothing is worse than this – if incorrect information is presented, users will not utilize it. Additionally, if a specific product has one price on the aggregator and changes upon visiting the seller's site, this also negatively impacts the project's reputation.
Solution: using residential proxies
Avoiding issues when scraping data for aggregation needs can be achieved by using residential proxies. Server IPs are provided by hosting providers. Identifying the ownership of an address by a specific provider is fairly simple – each IP has an ASN number that contains this information.
There are numerous services for analyzing ASN numbers. Often, they are integrated with anti-bot systems that block access for crawlers or manipulate the responses to their requests.
Residential IP addresses help bypass such systems. These IPs are assigned by internet providers to homeowners, with relevant markings in all associated databases. There are special residential proxy services that allow you to use residential addresses. – this is exactly such a service.
Requests sent by crawlers from residential IPs appear as if they are coming from regular users in a specific region. Regular visitors are not blocked – in the case of online stores, these are potential customers.
As a result, using rotating proxies from Infatica allows aggregator websites to obtain guaranteed accurate data while avoiding blocks and parsing difficulties.
Other articles on the topic of using residential proxies for business:
Source: habr.com
