Web scraping of prices
in retail

Price collection is the goal; web scraping is the method that automates it on a large scale. A price collection is only useful if it specifies which price it captures: store-specific price, card price, promotional price, or regular price.

When it comes to pricing, the GDPR is rarely the real legal issue. The limitations stem primarily from websites’ terms of use, database law, and the load placed on servers—as the CNIL itself points out.

Price monitoring is the process of observing the listed price of a product at a competitor’s store at a given moment. Web scraping is the method that automates this monitoring on websites and marketplaces on a large scale.

Our belief as retail pricing professionals: the most misleading price listing isn't the one that's missing—it's the one that doesn't specify which price is being referred to. Definition, how it works, and the legal framework.

Web crawler that extracts product prices and stores them in a structured database

A price survey involves observing and recording the displayed price of a product at a given moment at one or more retail locations. A category manager who notes the prices in a competitor’s aisle is already conducting a price survey.

Web scraping automates this data collection process. Bots query public web pages at scheduled intervals, extract prices, availability, and promotions, and store them in a usable database.

Data collection is the goal; scraping is just one method among many. The others include manual data collection—in-store or online—and market panels, which aggregate actual sales data with a lag of several weeks.

$1.56 billion

Estimated size of the global web scraping market in 2026, across all sectors. Price and competitor monitoring is the fastest-growing application from 2027 to 2032.
(Mordor Intelligence, Web Scraping Market, page updated on October 23, 2026)

It is no longer a DIY project reserved for advanced e-commerce merchants, but a component of the data infrastructure, with its own reliability requirements.

That's the question most systems don't ask. On the same website, a single product may be listed at multiple prices, but the bot only captures one of them.

Online prices often depend on the pickup store or the selected ZIP code. A pickup service displays the price for the selected store, not a national price: a comparison without a specific store to reference compares different items from one week to the next.

Add to that the price reserved for cardholders, the crossed-out promotional price, an offer from a third-party seller on a marketplace, and the bulk price. Each one tells a different story about the competitor’s positioning.

Our belief: A reading that does not specify which price it represents is more dangerous than no reading at all. It does not produce a visible error; rather, it produces a plausible discrepancy, based on which a decision is made.

Automated data collection is never just a single technology, but a three-layer process. The quality of the decision depends on the weakest layer, not the most impressive one.

1

Collection

Bots crawl product and category pages at a scheduled frequency on targeted retailers and marketplaces. The raw data does not yet have any decision-making value.

2

Product Matching

Each price collected is matched to the corresponding internal product record, even when the descriptions differ. This is the most critical step—and the one most often overlooked.

3

Refunds & Alerts

Significant discrepancies are identified and reported to the pricing teams, rather than getting lost in a raw data stream. This is where data becomes a decision.

Many online price monitoring projects focus all their energy on data collection (“more competitors, more frequently”) and neglect the next two layers.

One vulnerability goes unnoticed: indexing. Websites change their structure, strengthen their security measures, and some pages are no longer crawled without any warning.

A reliable system therefore measures the percentage of items actually scanned during each pass, not the number of competitors “tracked.” And it adjusts its frequency on a category-by-category basis, not uniformly.

Discussion with an Expert
Are your competitor price reports reliable?
Take 30 minutes to see how Booper collects and compares your competitors' prices— product by product —without any false discrepancies.
Let's plan an exchange →
SourceFreshnessCoverCostMain limitation
Manual Field SurveyWeekly to MonthlyLow, limited sample sizeHighDon't scale it beyond a few hundred references.
Internal scraping (DIY)DailyLarge, but fragileModerateOngoing robot maintenance + matching system to be developed
Specialized data scraping service providerReal TimeBroad, sharedSubscriptionReliance on a third party for reliability and support
Distribution panelsSeveral weeksVery large in actual volumeHighDoes not capture day-to-day price fluctuations

To put it this way: no single source is superior in every respect. Scraping supports day-to-day management, while the panel validates underlying trends. Details on these differences can be found in our article on scraping, field surveys, and panelists.

The question combines two different tasks. Collecting prices and determining a price are not handled by the same tool, and very few solutions do both well.

FamilyExamplesWhat they produceWhat They Don't Do
Generic ManifoldsBright Data, Oxylabs, Apify, Browse AILarge-scale page extraction from any websiteNo product matching, no price elasticity, and no price recommendations
Price Monitoring ToolsPrisync, Paarly, Minderest, Price2SpyA table showing competitors' prices for the products being trackedThe Decision: Margin, Sales Forecast, Distribution to Cash Registers
Pricing platformsBOOPER MPSOnline data collection, integration of store inventory reports provided by the retailer, product matching, elasticities, simulation, approved priceIt isn't marketed as a standalone scraping tool

The classic pitfall: buying a data collector, only to discover that the raw data can’t be used until after three projects that weren’t apparent at the time of purchase.

These areas of focus are product matching, consistency between online and in-store prices, and turning a price discrepancy into a decision. Dynamic pricing comes into play only at the very end: without these elements, it simply automates errors.

The selection criteria are detailed in our guide to choosing a competitor price tracking tool.

When it comes to price listings, the GDPR is rarely the real issue: a listed price is not personal data.

As the CNIL fact sheet itself points out, web scraping may be subject to other restrictions, such as the site’s terms of use, database producer rights, and copyright law.

Database law (Article L342-1 of the Intellectual Property Code) pertains to the extraction of a substantial portion of a database. Recording the prices of relevant items is not the same as copying an entire catalog, including descriptions and photos.

Our position: A defensible data collection process gathers only the minimum necessary information (price, availability, promotions, login credentials), respects websites’ opt-out signals, and spreads out its requests so as not to overload their servers.

This isn't just a legal precaution. It's also what makes data collection sustainable, because an aggressive bot ends up getting blocked. These principles are no substitute for legal advice regarding your system.

  • "Web scraping is illegal." Not true in principle: the collection of public data is regulated, not prohibited.
  • "The more competitors I have, the better my decisions are." This is false if product matching is unreliable. The choice of competitors is part of a strategy to monitor competitors' prices.
  • "A weekly report is sufficient." Not true for fast-moving categories: see the impact of up-to-date competitive data.
  • “Price tracking is already a strategy.” No: it’s input data, not a decision.

A survey is never an end in itself. It is the very first step in the pricing decision-making process: any error made at this stage ripples through every subsequent decision that depends on it.

At Booper

GENIUS Competitor receives price data collected online (websites, marketplaces, apps) and in-store or panelist data provided by the retailer. GENIUS Link links this data to your product listings using NLP, along with a confidence score for each product.

It is this system that enables a national food retailer to manage several million prices each year across a network of several hundred stores, ensuring consistent pricing nationwide.

Find out how Booper organizes this data stream on our price tracking and web scraping page, and how it fits into a comprehensive price monitoring system.

See also in this feature

The process of observing and recording a product's listed price at a given moment at one or more retailers. This can be done manually, in-store or online; automated through web scraping; or derived from panels that aggregate sales data.

The goal of price monitoring is to determine the prices charged by a competitor. Web scraping is one method for achieving this goal—and the only one that can scale beyond a few hundred product listings.

Yes, in principle: the collection of public data is regulated, not prohibited. When it comes to pricing, the risk stems less from the GDPR than from the websites’ terms of use, database law, and the load placed on servers.

On June 19, 2025, the CNIL clarified the safeguards required when personal data is involved.

There are three methods: manual price collection in-store, checking websites, and automated web scraping. In each case, make a note of which price you’re recording: the store’s regular price, the card price, the promotional price, or the standard price.

Booper collects prices online and incorporates in-store data or panelist data that the retailer obtains from other sources; it does not conduct in-store surveys.

It depends on the category, not on a single rule: daily—or even more frequently—for items with high promotional turnover, and weekly for items with stable prices. It all starts with choosing which products to monitor.

No, they complement each other. Market surveys track actual sales with a lag of several weeks, which is useful for identifying long-term trends. Web scraping provides listed prices in near real time, which is useful for making decisions this week.

A platform that combines data collection, product matching, and price recommendations into a single workflow.

BOOPER retrieves competitor prices from GENIUS Competitor, links them to your listings using GENIUS Link (NLP, confidence score), and then manages pricing—all without having to re-enter data between tools.

Sources: Mordor Intelligence, Web Scraping Market, page updated on October 23, 2026 · CNIL, Focus on Legitimate Interest: Data Collection Through Web Scraping, June 19, 2025 · Intellectual Property Code, Article L342-1.

‍

Articles
similar
A glass eye connected to satellite icons symbolizing the price monitoring system
August 27, 2026
Pricing Monitoring: The Complete Framework in 5 Components

A comprehensive competitor pricing monitoring system is built on five inseparable components: data collection, product matching, alerts, reporting, and governance. If even one of these components is missing, the system becomes ineffective.

Data collection is no longer the weak link: according to Diamart Group (2026), 94% of French retailers collect competitive data, but about half still perform the matching manually.

Read the blog post
A glass magnifying glass on a price tag: choosing a price-tracking tool
August 27, 2026
Monitoring Competitors' Prices: How to Choose the Right Tool

Coverage, timeliness, and compliance are prerequisites for a competitor price monitoring tool. Its value depends on the accuracy of product matching and on linking each price to a traceable decision, yet approximately half of French retailers still perform this matching manually (Diamart Group, 2026).

The listed cost is only part of the total cost. Manually reconciling discrepancies, maintaining in-house development, and making decisions based on unreliable data often end up costing more than the subscription.

Read the blog post
A store connected via a network of lines to a large-scale store network
August 16, 2026
How to Scale Up Competitive Intelligence on a Large Scale

A successful pilot test proves almost nothing: on a small scale, human attention masks the flaws in matching, alerts, and frequency that become immediately apparent on a large scale.

Systems that are in place advance in stages, but no stage is mandatory: we move on to the next one when maintaining the previous one costs more than moving beyond it. And what scales up is not the number of employees, but the traceability of decisions.

Read the blog post
Ready to
 booster
What about your profit margins?

The pricing intelligence solution for retail leaders. Accuracy, speed, and controlled margins.

Let's discuss your pricing challenges
‍
‍