What is the

price tracking or web scraping

in retail?

A precise definition, not a portmanteau

A price survey, strictly speaking, is the observation and recording of a product’s listed price at a given moment at one or more retailers. Nothing more. It’s a practice as old as commerce itself: a category manager who jots down the prices in a competitor’s aisle in a notebook is already conducting a price survey.

Web scraping is one of the methods for generating this report—the only one that can be scaled. Technically, it’s an automated process that queries public web pages (product listings, category pages, search results, marketplaces) at scheduled intervals, extracts the displayed data (price, availability, promotions, product descriptions), and organizes it into a usable format—a database, not a screenshot.

The confusion stems from the fact that “price monitoring” can refer to three very different approaches in terms of reliability, cost, and timeliness: field data collection (a person visits locations or checks prices manually), web scraping (a bot collects data automatically), and retailer panels (a third party aggregates data from point-of-sale systems or panelists).

$1.17 billion — estimated size of the global web scraping market in 2026, across all sectors, with adoption driven in particular by competitive intelligence and dynamic pricing (Mordor Intelligence, 2026).

It is therefore no longer a niche practice or a technical hack reserved for the most advanced e-commerce businesses: it has become a core component of data infrastructure, just like an ERP or a PIM, with its own reliability requirements.

How does automated price tracking work, technically?

Automated price discovery is never just a single technology: it is a three-layer pipeline, and the quality of the final decision depends on the weakest layer, not the most impressive one.

• Data collection. Robots crawl product and category pages at a scheduled frequency on targeted retailers and marketplaces. The raw data does not yet have any decision-making value.
• Product matching. Each price collected is matched to the corresponding internal product record, even when the descriptions differ. This is the most critical step—and the one most often overlooked.
• Reporting & Alerts. Significant discrepancies are identified and flagged for the pricing teams, rather than getting lost in a raw data stream. This is where data becomes a decision.

Many price monitoring projects fail because they focus all their energy on the first layer—“we’ll scrape data from more competitors, more often”—while neglecting the next two.

85% of executives believe their pricing decisions could be improved—but only 26% of companies use dedicated pricing software to make them more objective (Bain & Company / HBR.org, study of 1,700 companies, 2018).

It is precisely for this reason that a senior category manager must evaluate a price monitoring system based on the entire pipeline, not solely on the volume of data collected.

Four Ways to Create a Price Comparison: The Comparison Chart

To put it this way: none of these four sources is universally superior. A good architecture generally combines several sources depending on the use case: scraping for day-to-day price monitoring, and panel data to validate underlying trends.

The Five Criteria That Distinguish a Useful Report from a Stream of Raw Data

• A frequency appropriate for the category, not a one-size-fits-all frequency. A category with high promotional turnover warrants daily—or even multiple times a day—data collection.
• Actual coverage, not advertised coverage. A tool that “tracks 10 competitors” may fail to crawl 20% of the targeted pages without anyone realizing it.
• The quality of product matching. Linking a scraped SKU to its internal record is the most critical—and most underfunded—step in the process.
• Legal compliance of the system. Scraping public data is regulated, not prohibited in principle.
• Decision-oriented reporting. What saves time is a system that identifies significant discrepancies and alerts you to them.

An average increase of ≈8% in operating profit for every 1% improvement in price, assuming constant volume (McKinsey Quarterly, “Bringing Discipline to Pricing,” based on the S&P 1000, 2000).

Is price web scraping legal? What the current legal framework says

Legal Framework (CNIL, June 19, 2025). The CNIL acknowledges that data collection through web scraping of public data may be based on the legal grounds of legitimate interest, provided that three cumulative conditions are met: a genuine and specific purpose, the necessity of the processing to achieve that purpose, and a favorable balance between the company’s interests and the rights of the data subjects.

In practice, for standard B2B pricing monitoring, the legal risk remains manageable, provided it is properly documented—not ignored.

Four Common Misconceptions That Skew the Interpretation of a Price List

• "Web scraping is illegal." False as a general principle: the collection of public data is regulated, not prohibited.
• “The more competitors I have, the better I can make decisions.” This is not true if product matching is not reliable—the method for prioritizing competitors will be the subject of an upcoming article in this series, which focuses on developing a strategy for monitoring competitor prices.
• "A weekly report is enough." Not true for fast-moving categories—a topic we'll explore in an upcoming article on the impact of up-to-date competitive data on pricing decisions.
• "Price monitoring is already a pricing strategy." No: it's an input, not a strategy.

The price survey: the silent foundation of any pricing decision

A price survey is never an end in itself. It is the earliest input in any pricing decision—which means that any error made at this stage spreads, amplified, into every decision that depends on it.

At Booper, the GENIUS Link module uses NLP to automatically match your SKUs with those in competitors’ catalogs—with a confidence score for each matched product. Downstream, GENIUS Monitoring centralizes alerts so that the collected data can be turned into decisions. It is this comprehensive process that now enables Coopérative U to manage several million prices per year across more than 1,700 stores while maintaining consistent pricing nationwide.

Find out how Booper structures this entire chain on our price tracking & web scraping page.

Frequently Asked Questions

What is a price survey in retail?

This involves observing and recording the listed price of a product at a given moment at one or more retailers. It can be done manually (in-store, online) or automatically, via web scraping.

What is the difference between price tracking and web scraping?

Price monitoring is the goal (finding out the price at which a product is sold); web scraping is a method for achieving this on a large scale—an automated process that extracts prices from public web pages and organizes them into actionable data, without the need for repeated human intervention.

Is web scraping of competitors' prices legal?

Yes, in principle: collecting public data is not prohibited. The applicable framework depends on the nature of the data (whether it is personal or not), compliance with the terms of use of the website from which the data is collected, and the load placed on the server. In June 2025, the CNIL published specific criteria for grounding this type of data collection in legitimate interest.

How often should you check your competitors' prices?

It depends on the category: daily or even multiple times a day for items with highly volatile promotional prices, and weekly for items with stable prices. A uniform frequency across an entire catalog wastes data collection capacity without improving relevance.

Is web scraping replacing panels like NielsenIQ or Kantar?

No, the two sources are complementary. Retail panels aggregate actual sales data with a time lag, which is useful for validating underlying trends. Data scraping provides near-real-time data, product by product, which is suitable for daily pricing management.

Why is proper product matching essential for reliable price data?

Because a price comparison is only meaningful if you can be certain it corresponds to the exact same product as yours. Approximate matching leads to price differences that appear real but aren’t, and that silently skew all decisions based on them.

See also in this feature

• Scraping, field surveys, panelists: What are the differences?
• How to Develop a Strategy for Monitoring Competitors' Prices
• The Limitations of Excel for Monitoring Competitors' Prices

Sources

• Mordor Intelligence, Web Scraping Market Size & Share Report, 2026
• Bain & Company / HBR.org, “A Survey of 1,700 Companies Reveals Common B2B Pricing Mistakes,” June 2018
• McKinsey Quarterly, “Bringing Discipline to Pricing,” Winter 2000
• CNIL, The Legal Basis of Legitimate Interest — Data Collection via Web Scraping, June 19, 2025