Next Article How AI & proxies drive web scraping –

As public online data acquisition becomes increasingly important to decision-making, AI, web scraping and proxies will continue to find their way into business activities. While the inclusion of AI into web scraping is rather new, some data acquisition companies are already harnessing the power of machine learning.

In fact, proxies themselves are already being used in fast-growing industries like ecommerce and cybersecurity in one way or another, says Tomas Montvilas, the chief commercial officer (CCO) at Oxylabs, a proxy service provider:

“In short, proxies act as an intermediary that accepts connection requests from its user and sends them to a destination server. That means that servers – in most cases, plain old websites – think that the proxy is the original source of the request. In web scraping, proxies are mostly used for data request distribution and anonymity.

“There is no way to overstate the importance of proxies for certain business models. Some profit models rely on external data gathering (e.g. Semrush, who do SEO monitoring). These companies essentially sell data analysis software or the data itself.

“However, tried-and-true industries such as retail and financial services are beginning to incorporate public data gathering into their processes. Public data allows these businesses to gain a competitive advantage and drive additional growth.

“Proxies are a necessity for any business that wants to acquire high-quality public data. There are numerous ways they make the entire gathering process more reliable. Certain data is displayed differently based on the perceived location or device of the visitor (e.g. the price of an iPhone in the UK vs the price in Singapore). Proxies allow businesses to gather accurate information by harnessing the power of different IP addresses.”

Building blocks of web scraping

Public data gathering, on the face of it, is a rather simple process. An application goes through a list of URLs, downloads the data stored there, and eventually provides an output of everything that has been downloaded. 

Montvilas continues: “However, public data gathering processes need consistent access to accurate data. Different types of proxies help applications handle most aspects related to data access and accuracy. Businesses generally choose between residential or data centre proxies, depending on the data source, if they are looking for a simple solution.

“AI and machine learning-based solutions are still quite rare in the web scraping industry. Currently, machine learning is mostly being used to automate certain tricky processes where trial and error would otherwise be used. For example, with our Next-Gen Residential Proxy solution we have created AI-based models that greatly increase data acquisition success rates for our clients.”

There are many different proxy types used in web scraping activities. We asked Montvilas to describe the primary types and use cases for the different types of proxies in brief.

Residential proxies

“Residential proxies are the IP addresses of the computers, phones, or other devices granted by ISPs to regular customers. These devices become proxies whenever users install related software and consent to the related terms and services.

“We have sourced our 100 million+ residential proxy pool mostly by using a Tier A+ acquisition model. Put simply, it is the process of gaining IPs from consenting, aware users of a dedicated application and providing a monetary reward to them for any traffic use.”

Residential proxies are widely used by businesses that need rotating IP addresses and city-level targeting. “A part of our residential proxy users are ad verification businesses. Fighting against ad fraud means checking various websites from different locations and devices to determine whether ads are being displayed faithfully. Our development teams worked hard to provide global coverage and city-level targeting to our residential proxy pool, making it a great fit for ad verification businesses.

“We predict that proxy use for this business model is only going to increase from here onwards. An unfortunate reality is that ad fraud is on the rise. Predicted costs of ad fraud from 2018 to 2022 may rise from $19 billion to $44 billion. Residential proxies simply cannot be replaced by anything else, necessitating greater use over time if the trends continue. There are even businesses whose model is completely reliant on them. For example, Trivago, a renowned accommodation comparison service, needs residential proxies to accurately deliver location-based pricing.”

Next-Gen Residential Proxies

Next-Gen Residential Proxies are a unique product tied to Oxylabs themselves. Next-Gen Residential Proxies are an innovation in the industry by adding AI and machine learning to proxies.

“We developed Next-Gen Residential Proxies as an advanced version of residential proxies for those who are struggling with acquiring public data from complex targets. Our goal with Next-Gen Residential Proxies is to help businesses achieve 100 per cent data delivery success rates, making them perfect for targets with high failure rates such as ecommerce platforms.

Oxylabs fig 1 ProxyEgg Next Article How AI & proxies drive web scraping -
Source: Oxylabs


“We know that AI & ML have garnered a lot of hype in the IT sector over the recent years. However, hype means nothing if there are no results to show for it. Therefore, in order to ensure the success and effectiveness of our AI & ML innovation, we created an advisory board who guide us during our development processes. Our advisory board is composed of people who are actively involved in PhD level research on AI or are working with companies that are machine learning industry leaders.

“Next-Gen Residential Proxies are proof that AI and machine learning do have their place in public web scraping. Currently, our solution has two primary features that employ AI: dynamic fingerprinting and adaptive parsing. The former is an automated process that picks the best way to send an HTTP request to maximise success rates; the latter is the process of automatically structuring data found in ecommerce product pages and returning a structured result.”

Data centre proxies

Unlike residential IPs, data centre proxies are generally created by businesses that have access to reliable server infrastructure. Dozens of data centre proxies are borne out of one machine, making them a lot cheaper than their residential counterparts. Additionally, data centres have more reliable and faster internet connection than any device a regular consumer might have.

“Data centre proxies are the backbone of businesses that need to go through vast arrays of information on a daily basis. Data centre proxies are most commonly utilised in areas where access to data is not geographically restricted and traffic by IP is not as actively tracked. For example, brand protection companies comprise a large portion of our data centre proxy users.

“Performing daily brand protection activities (e.g. scanning the internet for counterfeit products) usually involve web scraping lots of data-heavy websites such as ecommerce platforms. Thus, using data centre proxies with the highest possible speeds and uptime is key to optimal business performance.”

Real-Time Crawler

Real-Time Crawler exists as an out-of-the-box solution for public data acquisition. Instead of developing a web data acquisition tool in-house and using proxies, Real-Time Crawler does everything outside of data analysis.

“While Real-Time Crawler is not a proxy, it utilises them to allow its users to perform their requests. Of course, we implement it with all the advancements made with AI and machine learning. For example, Real-Time Crawler takes advantage of AI-powered dynamic fingerprinting, just like Next-Gen Residential Proxies.

“As a solution, Real-Time Crawler can be considered as a data API. Users can use highly customisable HTTP requests to scrape data according to their needs. These requests can contain many different parameters, such as proxy location, device, result language, etc.”

All types of businesses use Real-Time Crawler as their primary source of external web data, including any business that needs to monitor search engines, ecommerce platforms, or other websites.

“In ecommerce, data acquired from Real-Time Crawler is often used for pricing tracking and analysis, modelling market trends, and doing platform-specific keyword research. Real-Time Crawler is tailored for those businesses that want to quickly kickstart their public external data gathering without the hassle of managing and maintaining gathering tools.

“Use cases with search engines vary but most are heavily related to SEO. Predictions about optimisation can often be made only with the help of reverse engineering ranking algorithms from data, making Real-Time Crawler a candidate for some SaaS businesses in the SEO industry.”

Rising tides in the proxy industry

Proxies are here to stay. With the Covid pandemic accelerating the movement from retail to ecommerce for nearly all businesses, the proxy traffic per day is projected only to rise from here onwards.

“Our internal data reveals a meteoric rise of proxy traffic use in Q4 of 2020 alone. During Q4, traffic use increased to previously unseen heights. For example, on Black Friday residential proxy traffic shot up by 301 per cent, while data centre proxy traffic rose by 97 per cent compared to the same period in 2019. Additionally, surges in traffic use rose a week in advance of Black Friday in 2020, compared to a day [in advance] in 2019. Therefore, as we can clearly see, more and more companies are getting involved in public data gathering in order to stay relevant and attain profitable insights.

“Enquiries regarding various ecommerce and scraping aspects, including some well-known names in the industry, rose exponentially over the past year. While Real-Time Crawler hasn’t struggled to meet demand, it has been stress tested numerous times by the rising need of data.”

Web scraping and proxy use is expected to continue to rise as businesses want to unlock the insights provided by online public data. As AI and machine learning become increasingly popular, the effectiveness of external data acquisition is only going to increase. Businesses that want to keep raising profits will need to, in one way or another, implement public data gathering and analysis.

Tomas Montvilas
Tomas Montvilas

Tomas Montvilas is a chief commercial officer at Oxylabs, a leading big data infrastructure and proxy solutions provider. He is an expert of organisational growth with over seven years of experience in leadership roles in the areas of sales, marketing, product development and digital transformation.

Source of this news:

Related posts:

Netflix Networking: Beating the Speed of Light with Intelligent Request Routing -
Transcript Fedorov: This presentation is about improving performance of network requests. It's been known for years that latency of network interactions has large impact in many business areas. For e...
The world's worst kept secret and the truth behind passwordless technology - Help Net Security
One of the biggest security risks of modern-day business is the mass use of passwords as the prime authentication method for different applications. When the technology was first developed, passwords...
'I let the community down' viewpoint Kaseya CEO explains the server restart was spurred back followi...
Kaseya's CEO has apologised to its customers punch by last week's cyber-attack and says the restart from the VSA servers has been stressed back until Sunday to enable extra security measures to g...
Assay for the sensitive and specific identification of SARS-CoV-2 Delta variant - News-Medical.Net
The Delta variant (B.1.617.2) of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was first detected in India and is currently replacing the variants circulating in Europe, the USA, and m...
How to set up a proxy server in Edge for Windows 10 - Windows Central
In a time of restrictions and eroding privacy, many people are using a proxy while they browse the internet. A proxy is essentially a secondary hub that your internet traffic is pushed through. Inste...
Metabolic differentiation and intercellular nurturing underpin bacterial endospore formation - Scien...
Experimental methodsStrain construction. All the strains used in this study are derivatives of B. subtilis PY79. A complete list of strains is provided in table S2. The plasmids and oligonucleotides ...
Fix Screen Rotation Issues on Yoga 2, 3 Pro in Windows 10 - Windows Report
by Ivan Jenic Troubleshooting Expert Passionate about all elements related to Windows and combined with his innate curiosity, Ivan has delved deep into understanding this operating syste...
Pfizer, Moderna expand studies from COVID-19 vaccine to offspring age 5 to 22 - Baltimore Sun
Multiple citizens familiar with the trials menti one d the Food and Drug White house has indicated to Pfizer-BioNTech and Moderna that the scale and scope of their pediatric studies, as initially...
Monetizing email ads will be difficult on iOS 15 -
“”Sell ​​cider“” Is a column written by the sellers of the digital media community. Today’s column is written by Chris Suptoline, Vice President of Marketing at Kebel. With the official release of i...
UMass Memorial notifies 209K patients 8 months after data breach discovery - SC Magazine
When a breach attack affects one or two organizations — especially financial institutions or other businesses in highly regulated industries, which hold oodles of sensitive information — it can be ba...
How to work with user classes on Windows 2021 Tips - Bollyinside - BollyInside
This tutorial is about the How to work with user classes on Windows. We will try our best so that you understand this guide. I hope you like this blog How to work with user classes on Windows. If you...
Five reasons to use residential proxies for web scraping - Tech Digest
Residential proxies are one of the most creative and efficient tools you can have for your company’s digital toolbox. People who require scraping the web for the business need residential proxies the...
LRRC8A-containing chloride channel is crucial for cell volume recovery and survival under hypertonic...
The regulation of cell volume is essential for organism homeostasis (1). Cell swelling or shrinkage following osmotic stress exerts profound alterations of the cellular status (2), from short-term ch...
TheSocialProxy Review: Taking Social Media Management to the Next Level - Make Tech Easier
As a social media marketer, or simply a person who manages multiple social media accounts, you may benefit from using a proxy service. Most social networks don’t allow multiple accounts, so the...
Maryland’s enrollment in Obamacare to our lives 12% during coronavirus pandemic emergency - Baltimor...
This state’s health emergency, in regards to 54, 402 people enrolled in federally subsidized private blueprint offered by three private insurance companies, while 20, 460 signed up without financ...
Error Writing Proxy Settings, Access is denied in Windows 11/10 - TheWindowsClub
After you log in to your Windows computer or execute a command in Command Prompt or Windows Terminal, you may receive a message — Error Writing Proxy Settings, Access is denied. This error occurs if ...
Knicks Morning News (2022. '04. 08) – KnickerBlogger. Hook - KnickerBlogger
Knicks vs . Wizards: Think about time, where to watch, something the latest – Hoops Build up [] — Friday, The spring 8, 2022 3: 32: 54 AM Knicks vs . Wizards: Start valuable time...
New SideWalk Backdoor Targets U.S-based Computer Retail Business - The Hacker News
A computer retail company based in the U.S. was the target of a previously undiscovered implant called SideWalk as part of a recent campaign undertaken by a Chinese advanced persistent threat group p...

IP Rotating Proxy Onsale


First month free with coupon code FREE30