Project overview
The challenge
- Leboncoin.fr implements sophisticated anti-scraping protection that blocks automated requests and bot traffic, making traditional scraping approaches ineffective.
- Phone numbers are only revealed after JavaScript execution and user interaction, requiring full browser rendering capabilities rather than simple HTML parsing.
- Access to local listings requires French IP addresses due to geolocation restrictions, creating barriers for international scraping operations.
- Frequent HTML structure changes and dynamic CSS classes require robust extraction strategies to maintain reliability over time.
- Need to balance extraction success rate with API costs for scalable operations, optimizing the cost-per-contact metric.
Research project exploring modern bot-detection architectures by building a structured data extractor targeting Leboncoin.fr, France's largest classified-ads platform.
Implemented ScrapFly's residential proxy network with French geolocation and JavaScript rendering, demonstrating how dynamic content protection and geolocation restrictions interact with automated clients.
Designed a robust 3-tier fallback extraction strategy combining HTML selectors, link detection, and regex patterns to handle frequent HTML structure changes — a common challenge in production data pipelines.
Optimised for cost efficiency with configurable limits and environment-based configuration, demonstrating $0.33 per 10 records while maintaining high success rates across light to heavy usage patterns.
Technical architecture
ScrapFly Integration Layer: Handles API authentication, anti-scraping protection bypass, and residential proxy configuration with French geolocation
Two-Phase Scraping Pipeline: Phase 1 collects ad URLs from search results, Phase 2 extracts detailed information including phone numbers
Multi-Method Phone Extraction: Implements 3-tier fallback strategy using HTML selectors, phone links, and regex patterns with French format validation
Data Processing Pipeline: BeautifulSoup HTML parsing with robust price extraction handling Unicode spaces and structured JSON output
Key challenges & solutions
Anti-Scraping Detection
Implemented ScrapFly's ASP bypass with residential proxy rotation, realistic browser headers, and auto-retry mechanism to avoid detection and blocking.
JavaScript-Rendered Content
Enabled full JavaScript execution with 3-second rendering wait, DOM readiness checks, and auto-scroll to trigger lazy-loaded phone numbers.
Dynamic HTML Selectors
Built 3-tier fallback extraction using data-qa-id attributes, tel: links, and regex patterns to handle frequent HTML structure changes.
Cost Optimization
Implemented configurable limits, environment-based tuning, and efficient two-phase approach to minimize API costs while maintaining quality.
Data Validation
Created comprehensive validation system with regex patterns, length checks, prefix validation, and duplicate detection for data quality.
Impact & results
Achieved 100% phone extraction success rate in production tests (10/10 contacts)
Optimized performance to 60 seconds execution time for 10 contacts
Reduced cost to $0.33 per 10 contacts through efficient API usage
Enabled scalable operations from $4.50/month (light) to $30/month (heavy usage)
Open-sourced with MIT license, comprehensive documentation, and validation reports
Key features
- Bot-detection architecture research using ScrapFly ASP
- Residential French proxies demonstrating geolocation-based access patterns
- JavaScript rendering for dynamically-loaded content
- Multi-method extraction with 3-tier fallback strategy
- Robust price parsing with Unicode space handling
- Environment-based configuration (.env support)
- Comprehensive error handling and logging
- Production-validated with detailed test reports
