Price List - Web Scraping
On this page
Price monitoring involves collecting the prices charged by your competitors for a sample of products, either in-store or online. It can be done manually (by field researchers) or automatically (via web scraping, APIs, or data panels). It is the first step in any price monitoring strategy.
The Essentials in 6 Questions
Collect competitors' prices, promotions, and availability, product by product.
Pricing teams, category managers, sales departments.
Ongoing:daily for e-commerce, weekly for most brick-and-mortar departments.
E-commerce sites, marketplaces, and local stores.
Find out whether you're competitive or out of the market with your key products.
Field surveys, automated web scraping, or data panels.
Why a price list is essential
Without a price survey, it's impossible to know where you stand in the market or where to focus your efforts first.
- Understand your market position: determine whether you are competitive or out of the market for your key products, starting with your KVI.
- Identifying opportunities: Regular monitoring reveals where you can raise your prices (when competitors are more expensive) and where you need to adjust them (when competitors are aggressive).
- Feeding data into analytical tools: The data feeds into your price index, dashboards, and price optimization software.
A concrete example of a price list
A home improvement store monitors 300 products from five competitors every week and identifies 40 products that are overpriced.
Collected weekly: shelf prices and end-cap promotions.
competitors in the catchment area, monitored on an ongoing basis.
identified as being 10% more expensive than the competition, and then adjusted.
Case Study: Booper Pricing Glossary.
Investigators record the prices displayed on store shelves and any promotions. The data is compiled in a dashboard that automatically calculates price discrepancies on a product-by-product basis, enabling the pricing team to decide which SKUs to adjust. In-store price checks are legal as long as they are limited to simple observation (seethe National Consumer Institute).
How do you conduct price surveys?
There are three methods; the most effective one combines web scraping for scalability and human review for ambiguous cases.
| Method | Strengths | Limits |
|---|---|---|
| Field Survey | Reliable and essential for brick-and-mortar stores and products not available online. | Expensive and slow. |
| Web scraping | Fast, comprehensive, and affordable: real-time pricing, inventory, and promotions. | Requires rigorous product matching. |
| Data Panels | Accurate figures based on sales data from several retailers. | Limited to participating retailers. |
A modern price report is no longer limited to the listed price: it also includes promotions, crossed-out prices, shipping costs, availability, and data from marketplaces. Booper automates these competitor price reports through web scraping—including quality control—and then consolidates them to help you manage your pricing strategy. The price report is thus the first step in competitor monitoring based on product matching.
The 3 Mistakes That Skew a Price Survey
A survey is valuable only if it is consistent, representative, and complete.
- Irregular data collection: A data collection every 3 months isn't enough in e-commerce, where prices change every day. Adjust the frequency to match the pace of the market.
- Non-representative sample: Selecting 20 random products is meaningless. Focus on your KPIs and your high-turnover products.
- Ignoring promotions: Comparing "regular" prices without taking current promotions into account gives a distorted view of the competition.
Frequently Asked Questions
Short answers to the most frequently asked questions about the price list.
What is a competitor price survey?
This involves collecting and analyzing the prices charged by players in your market to assess your pricing position, monitor your price-sensitive products, and respond to competitors' moves.
How can you automate the collection of competitor prices?
Through web scraping: bots collect prices, promotions, and product information from competitors' websites. This is the principle behind online price monitoring, which replaces manual price checks—which are time-consuming and quickly become outdated—provided it is paired with reliable product matching.
What data should be collected during a price survey?
The listed price, as well as promotions, crossed-out prices, shipping costs, availability, product specifications, and marketplace data.
What is the difference between a price-tracking tool and pricing software?
The data collection tool gathers and presents competitive data. The pricing software transforms this data into recommendations and decisions, with impact simulations, business rules, and validation.
How often should prices be updated?
Daily for e-commerce, weekly for most in-store categories, and with a focus on your KPIs.
Key Takeaways
- The price survey measures your actual positioning relative to competitors, both in-store and online.
- Web scraping provides the scale, fieldwork covers the physical aspect, and product matching ensures accuracy.
- Track your KPIs regularly as a priority, and always include promotions in your calculations.
Would you like to automate your price tracking?
Reliable, up-to-date competitive prices—both in-store and online—without the hassle of gathering them yourself.
Let's talk about your price quotes →Discover our price monitoring softwareYou might also be interested in these articles
Effective pricing management requires the rigorous integration of internal/endogenous data (costs, historical data) and external/exogenous data (competition, demand). This essential hybridization secures margins and objectifies trade-offs against market fluctuations. By structuring these signals, the organization transforms raw data into an operational profitability lever, deployable in practice in less than sixty days.
Price monitoring is the goal—finding out at what price a product is sold by competitors; web scraping is the method that makes it possible to achieve this on a large scale, through an automated process rather than manual data collection.
The global web scraping market is estimated at $1.17 billion in 2026, driven notably by competitive intelligence and dynamic pricing (Mordor Intelligence).
In-store surveys, web scraping, and retailer panels each answer a different question: what the customer sees on the shelf, what is displayed online at a given moment, and what has actually been sold. Confusing them is like answering the wrong question with the right data.
Among large multichannel retailers, the proportion of prices that change each month rose from 15% to nearly 30% between 2008 and 2017—a pace that a weekly survey or monthly panel is structurally unable to keep up with (Alberto Cavallo, NBER).