Scraping, field surveys, panelists: What are the differences?
In-store surveys, web scraping, and retailer panels each answer a different question: what the customer sees on the shelf, what is displayed online at a given moment, and what has actually been sold. Confusing them is like answering the wrong question with the right data.
Among large multichannel retailers, the proportion of prices that change each month rose from 15% to nearly 30% between 2008 and 2017—a pace that a weekly survey or monthly panel is structurally unable to keep up with (Alberto Cavallo, NBER).
“We’re already monitoring our competitors” means nothing until we’ve answered a specific question: What data, collected how, and how up-to-date? Three very different practices lie behind this statement (field surveys, web scraping, and retailer panels), and each answers a different question, with costs, timeliness, and reliability that are worlds apart. Confusing them isn’t just a methodological nuance—it means making a pricing decision based on the wrong data source. This guide compares the three methods point by point and provides the criteria for building an architecture that combines them, rather than choosing just one by default.
Why These Three Methods Are Consistently Confused
In most retail organizations, “price monitoring” encompasses three distinct processes. Field data collection: a person observes and records a price, either in person or by manually checking it. Web scraping: A bot automatically queries web pages and organizes the displayed prices into actionable data; our definition article explains this mechanism in detail. Retail panels: A third party aggregates data from point-of-sale systems or panelists—acknowledging a time lag—to measure actual demand.
Treating these three sources as interchangeable comes at a high cost—in very practical terms. A team that bases its weekly adjustments on panel data received several weeks late will always be reacting after the market has already moved. Conversely, a team that bases a strategic repositioning on three days of scraping data will mistake statistical noise for a long-term trend, due to the lack of historical depth provided by a panel.
Field Survey
A person walks around or checks out the store, noting what they see: listed prices, as well as merchandising and actual out-of-stock items.
Web scraping
A bot automatically scans product pages and organizes the displayed prices into actionable data.
Distribution panels
A third party aggregates data from point-of-sale systems or panelists—with a time lag—to measure what was actually sold.
Choose based on the question asked
Each method answers a different question. Confusing them means answering the wrong question with the right data.
The following three sections explain, one by one, what each method actually captures—and, more importantly, what it does not capture.
Field Surveys: What They Capture, What They Don't Capture
Field data collection involves observing and recording—either in person or manually—the displayed price of a product at a retail location, such as a category manager surveying a competitor’s aisle, a commissioned researcher visiting a retail location, or an employee manually noting prices displayed online.
This is the only one of the three methods that captures what is actually happening on the shelf: product availability as seen by the customer—not the availability listed online—merchandising execution, consistency between the price advertised at the end cap and the price actually scanned at the register, and the visual impression a shopper has at the moment of decision. No web scraping tool, no matter how sophisticated, can detect an empty shelf or a poorly labeled promotion, because a website may list a product as “available” even though it has been physically out of stock for two days.
This is the average out-of-stock rate observed in stores among a global sample of more than 71,000 consumers in 29 countries—a figure that no online data can reveal (Corsten & Gruen, Harvard Business Review, 2004).
Its limitation is structural, not technical: it doesn’t scale. A field survey of 200 SKUs across 5 retailers requires a fixed amount of man-hours; multiplying that by 10 categories and 20 retailers amounts to funding a dedicated full-time team, with costs that rise linearly while those of web scraping barely increase. For this reason, in most mature systems, fieldwork remains a targeted supplement (a one-time audit, sample-based quality control) rather than the foundation of day-to-day pricing management.
Web scraping: Freshness and granularity, but specific blind spots
Web scraping—whose three-layer technical process (collection, matching, and output) is detailed in our definitional article—is an automated process that queries public web pages at scheduled intervals and organizes the displayed prices into actionable data. Its strength lies in two aspects: timeliness, with the ability to collect data daily or even multiple times a day, and granularity product by product, without any scaling limitations imposed by human time constraints.
This freshness is not a luxury; it is a necessity dictated by the market itself.
This is the proportion of prices that change each month at major multichannel retailers, which rose from 15% in 2008–2010 to nearly 30% in 2014–2017—a pace that a weekly survey or monthly panel is structurally unable to keep up with (Alberto Cavallo, NBER, Working Paper 25138, 2018).
Web scraping has specific blind spots, mirroring those of the physical store. It only sees what a website publishes—never the physical store itself, nor the actual price paid at the register after a discount not listed online, nor the actual availability on the shelves. It also depends entirely on the structure of the targeted site: a change in the template, enhanced anti-bot protection, or a page that switches to client-side dynamic rendering can interrupt data collection without anyone noticing, if the coverage rate isn’t continuously monitored.
Distributor Panels: The Truth Delayed
Retail panels (NielsenIQ, Kantar Worldpanel, and their industry-specific equivalents) do not measure the listed price: they measure the price actually paid, aggregated from checkout data or receipts reported by a sample of households. This is a difference in nature, not in degree, from data scraping. A panel tells us what was sold, at what price, and in what volume. Web scraping tells us what is displayed at a given moment, without any certainty as to what was actually purchased at that price.
The price of this reliability is a structural time lag. Data from a panel is collected, aggregated, deduplicated, and validated before being reported—a process that, depending on the provider and the market, can take several weeks.
stores worldwide, tracking 50 million products each month: the scale of the retail panels provides a comprehensive and reliable picture of actual demand, but at the cost of a monthly reporting cycle that is incompatible with responsive pricing management (NielsenIQ, Retail Measurement Services).
This time lag makes a panel unusable for reactive price adjustments, but that is precisely what makes it irreplaceable for confirming a long-term trend. Data scraping may indicate that a competitor has been displaying a price 5% lower for the past three days; only a price panel can confirm, several weeks later, whether this displayed price difference resulted in an actual shift in volume or remained a display artifact with no measurable impact on demand.
A Detailed Comparison of the Three Methods
| Method | Freshness | Product Granularity | Cost | What Is Actually Measured |
|---|---|---|---|---|
| Field Survey | Punctual | Limited quantity | High | Perception, merchandising, actual shelf availability |
| Web scraping | Real Time | Very thin, product by product | Moderate | Price displayed online at this moment |
| Distribution panels | Several weeks | Grouped by category / reference | High | Price actually paid, volume sold |
To put it this way: the “what is actually measured” column is the most important of the four. A scraped listed price, a price paid as measured by a panel, and a perception observed in the field are not three more or less accurate versions of the same information; they are three different pieces of information that answer three different questions. Confusing the columns means answering the wrong question with the right data.
How to Combine Them in a Hybrid Architecture Based on Your Maturity Level
The question is never “scraping, fieldwork, or panel,” but rather in what order and for what purpose to combine them. The maturity of a price monitoring system is built in stages, not through a binary choice made once and for all.
One-time assignment + spreadsheet
Initial readings were recorded manually and compiled by hand, often in a spreadsheet. This method works on a small scale but cannot handle a larger volume of data.
Single-source scraping for day-to-day management
Automated scraping takes over for the most volatile categories, with product matching that is still rudimentary.
Multi-source scraping + targeted fieldwork for quality control
Matching is becoming more reliable, coverage is expanding, and an occasional field survey verifies what data scraping cannot detect.
Governed architecture, organized by category
Data scraping, field research, and panel surveys coexist, each assigned to the decision it best informs, with a governance structure that resolves any discrepancies.
At each stage, the question that should guide the decision isn’t “which method is best” in absolute terms, but rather which benchmarks and categories justify the investment—a detailed prioritization covered in our article on the products you should really be monitoring. Many organizations remain stuck at the first stage, using a system cobbled together in a spreadsheet, for reasons documented in our article on the limitations of Excel for price monitoring.
At Booper
This is exactly the path taken by the Barbotteau Group, a leading retailer in the French Caribbean with more than 70 companies and brands: a management approach historically built on Excel, followed by a rules-based system, then hybrid AI, and finally full automation—each stage tailored to the organization’s specific needs. After this data is collected—regardless of the source—the GENIUS Link module automatically matches SKUs using NLP, with a confidence score for each product, and GENIUS Monitoring centralizes alerts so that each source informs a decision rather than adding to yet another spreadsheet.
The right question isn't "which one to choose"
None of the three methods is universally superior to the other two; each answers a question that the other two cannot ask in its place. Field research answers, “What does the customer see and experience on the shelf?” Web scraping answers, “At what price is my competitor currently listing this product?” The panel answers, “What actually sold, and at what price?” Find out how Booper structures this comprehensive architecture on our price tracking & web scraping page.
A Checklist to Review Before Evaluating Your Price Monitoring Architecture
- Do you know which decision each data source is meant to inform, or do you use them interchangeably?
- Does your scraping process capture the up-to-date information needed for your most volatile categories?
- Does a field survey—even a one-time one—verify what web scraping can't detect (actual stockouts, merchandising)?
- Is your panel data used to validate a trend or for day-to-day management? This is a common point of confusion.
- Does your system combine these sources, or does it use only one by default due to a lack of thoughtful architecture?
FAQ
The questions we're asked most often before getting started.
Field data collection involves observing and recording—either in person or through manual data entry—the displayed price of a product at a retail location, such as a category manager surveying a competitor’s aisle or a researcher visiting a retail outlet. It is the only method that captures what is actually happening on the shelf: product availability as seen by the customer, the execution of merchandising, and consistency between the price advertised at the end cap and the price scanned at the register.
Web scraping, on the other hand, is an automated process that queries public web pages at scheduled intervals and organizes the displayed prices into usable data, without any limitations on scale due to human time constraints, and with data freshness that can be as up-to-date as real time.
Its counterpart is a blind spot that is symmetrical to that of the store floor: scraping never detects an empty shelf, a poorly marked promotion, or online availability that does not reflect the physical reality of the store. A website may list a product as “available” even though it has been out of stock for two days.
For this reason, in mature systems, fieldwork remains a targeted supplement (ad hoc audits, sample-based quality control) rather than the foundation of day-to-day management, due to the inability to scale it up: a survey of 200 SKUs across five retail chains already requires a significant amount of time that cannot be reduced.
No. A retailer panel does not measure the listed price, but rather the price actually paid, aggregated from checkout data or receipts reported by a sample of households—a difference in nature, not in degree, from data scraping.
The price of this reliability is a structural time lag: data is collected, aggregated, deduplicated, and validated before being reported—a process that, depending on the provider and market, can take several weeks. NielsenIQ, for example, tracks approximately 900,000 stores worldwide for 50 million products each month—a massive scale, but at a monthly frequency that is incompatible with responsive pricing management.
This time lag makes a panel unusable for day-to-day price adjustments, but that is precisely what makes it irreplaceable for confirming a long-term trend: data scraping may indicate that a competitor has been offering a price 5% lower for the past three days, but only a panel will confirm—several weeks later—whether that price difference has resulted in an actual shift in sales volume.
A comprehensive price monitoring system therefore requires both approaches: web-based price monitoring (scraping) for day-to-day responsiveness, and a panel for in-depth validation.
No, it captures the price displayed at a given moment, not necessarily the price actually paid. A discount applied only at the register, a negotiated price, or a loyalty program benefit not reflected online are not visible through web scraping.
That is precisely the role of retailer panels: they measure the actual price paid and the volume sold, aggregated from point-of-sale data or consumer panels, whereas web scraping only captures what a website publishes.
Web scraping also has blind spots that depend on the technical structure of the targeted site: a change in the template, enhanced anti-bot protection, or a page that switches to client-side dynamic rendering can interrupt data collection without anyone noticing if the coverage rate isn’t continuously monitored.
That is why the listed scrap price, the price paid as measured by a panel, and field observations are not three more or less accurate versions of the same information: they are three different pieces of information that answer three different questions.
Combine them. Data scraping supports daily pricing management thanks to its up-to-date nature; field staff periodically check for what the scraping process cannot detect (actual out-of-stock situations, merchandising); and the panel verifies that the observed discrepancies actually translate into real volume.
The right balance is achieved through stages of maturity rather than a one-time, binary choice: starting with ad hoc fieldwork compiled in a spreadsheet, then moving to single-source scraping of the most volatile categories, followed by multi-source scraping with fieldwork for quality control, and finally arriving at a governed architecture where all three sources coexist, each assigned to the decision it best informs.
The question that should guide the decision is never “which method is best” in absolute terms, but rather which benchmarks and categories justify investing in each one.
This is exactly the path taken by the Barbotteau Group, a leading retailer in the French Caribbean with more than 70 companies and brands: a management approach that historically relied on Excel, then moved to a rules-based system, followed by hybrid AI, and finally full automation—with each stage tailored to the organization’s specific needs.
Because their data is derived from the collection, aggregation, deduplication, and validation of point-of-sale data or large-scale consumer panel data—a process that, to remain statistically reliable, cannot be instantaneous.
Unlike web scraping, which structures data collected directly from a web page, a panel must gather data from thousands—or even millions—of retail locations or households before it can produce a representative figure. NielsenIQ, for example, tracks approximately 900,000 stores worldwide for 50 million products each month—a scale that explains why data is collected monthly rather than daily.
This delay is not a flaw to be corrected; it is the trade-off for the reliability we seek: a panel provides data on what was sold, at what price, and in what volume, with a level of statistical rigor that an instantaneous data collection could not guarantee.
This is what makes a panel suitable for validating underlying trends, but structurally unsuitable for day-to-day, reactive pricing management—a role that is better fulfilled by web scraping.
Yes, within a targeted area. Web scraping never captures an empty shelf, a poorly labeled promotion, or the actual customer experience on the sales floor—elements that only an on-site survey, even an occasional one, can objectively document.
According to a Harvard Business Review study, the average out-of-stock rate on store shelves, based on a large international sample, is 8.3 percent —a figure that no online data can reveal, since a website may show a product as available even though it has been physically out of stock for several days.
The field therefore retains its role—no longer as the foundation for day-to-day management, though its scaling limitations remain structural—but as a targeted supplement: an ad hoc audit or sample-based quality control check designed to verify what scraping is structurally unable to capture.
In a mature price monitoring architecture, these three sources (field data, web scraping, and panel data) coexist, with each one assigned to the decision it best informs, rather than being treated as interchangeable.
See also in this feature
- What is price tracking or web scraping in retail?
- Competitor Prices: Which Products Should You Really Keep an Eye On?
- The Limitations of Excel for Monitoring Competitors' Prices
Sources: NBER, Alberto Cavallo, “More Amazon Effects: Online Competition and Pricing Behaviors,” Working Paper 25138, 2018 · Corsten & Gruen, “Stock-Outs Cause Walkouts,” Harvard Business Review, May 2004 · NielsenIQ, Retail Measurement Services.
A comprehensive competitor pricing monitoring system relies on five inseparable components: data collection, matching, alerts, reporting, and governance; if even one of these components is missing, the system becomes ineffective. The retail sector revises its prices more frequently than any other (ranging from monthly to daily, depending on the category), which requires a system capable of keeping pace with this frequency.
Seven criteria distinguish a competitor pricing tool that simply generates a table of price differences from one that actually drives decisions: coverage, timeliness, product matching, alerts, governance, integration, and compliance. The listed cost is only part of the total cost: manual reclassification, maintenance of in-house development, and the opportunity cost of a poorly informed decision often weigh more heavily than the subscription fee.
A price monitoring system that works perfectly in a pilot program does not deliver the same results on a large scale—not because the technology itself changes, but because the scale changes the nature of the problems to be solved: approximate matching, unmanaged alerts, and poorly calibrated data freshness become apparent where human oversight previously masked them on a small scale.
Only 26% of companies move beyond the proof-of-concept stage to derive tangible value from AI (BCG, 2024)—a reminder that the implementation of a pricing monitoring system depends first and foremost on methodology and governance, not on technology alone.