PriceWatch Scraper
Distributed price monitoring system scraping 50 competitor websites and 10,000 product prices daily — with proxy rotation, delta detection, and webhook delivery.
Client background
RetailEdge is a Cologne-based e-commerce analytics company that provides competitive intelligence tools to mid-size online retailers in the DACH region. Their flagship product is a price comparison dashboard — but the data feeding it was collected manually by a full-time analyst.
The challenge
One analyst was spending 8 hours per day manually visiting 50 competitor websites, copying prices into a spreadsheet, and uploading the CSV to a dashboard. This allowed only a once-per-day refresh and covered fewer than 5% of the products their clients wanted to track. Two competitors had already launched automated solutions; RetailEdge was losing clients who wanted real-time data.
Discovery & planning
Site Inventory Review
Client provided a spreadsheet of 50 target websites. We audited each for anti-bot protections: 32 used basic headers, 12 used Cloudflare, and 6 used advanced JS fingerprinting challenges.
Anti-Bot Assessment
We ran test scrapers against all 50 sites. 38 could be scraped with standard Playwright. 12 required residential proxy rotation. Shared detailed report with bypass strategy per site.
SLA & Frequency Agreement
Client needed a 6-hour refresh cycle (4× per day). We agreed on a 95% success rate SLA — defined as ≥95% of target product pages returning a price per cycle.
Delivery Format Design
Designed the PostgreSQL schema and webhook payload format. Client's existing dashboard would consume a REST API endpoint — we delivered the spec and they confirmed compatibility in day 3.
Technical solution
We built a distributed scraping system using a Redis job queue that dispatches scrape tasks to a pool of Playwright workers running in Docker containers. Jobs rotate between residential (Bright Data) and datacenter proxies based on the target site's protection level. Completed prices are compared against the previous run — only changed prices trigger a webhook to RetailEdge's dashboard. A monitoring dashboard shows job health, proxy usage, and per-site success rates.
Tech stack
Timeline
Playwright scrapers for all 50 sites, proxy rotation setup, Bright Data account config, initial accuracy tests
Redis job queue, Docker containerisation, AWS EC2 deployment, auto-scaling config, monitoring dashboard
Price comparison logic, webhook delivery system, FastAPI endpoint, client dashboard integration testing
72-hour full-cycle test run, SLA verification (97% success rate achieved), documentation, analyst handover
Results & outcomes
We now have real-time competitive pricing intelligence that used to take a full work day to compile. The delta detection was a brilliant touch — our dashboard only updates when something actually changes, so our analysts trust the data immediately.
Book a free call and we'll scope your project in 30 minutes.
Book a free call