Dubai Lead Generation Platform: Automated CRM Data Processing
Python/Flask lead management system with CSV processing, phone validation, Twilio SMS integration, and PostgreSQL for Dubai real estate market.
Project Overview
The Challenge
Real estate agents received leads from 10+ sources in inconsistent formats, requiring hours of manual cleanup.
Purchased lead lists contained 30-40% invalid phone numbers, wasting SMS campaign budgets.
No centralized system to track lead sources, making it impossible to measure ROI on marketing channels.
Manual SMS outreach was time-consuming and inconsistent.
Engineered a specialized lead generation and management platform for the Dubai real estate market, processing thousands of leads from various sources with automated data cleaning and enrichment workflows.
Built ETL pipelines using Pandas to validate, deduplicate, and normalize lead data from CSV imports. Implemented international phone number validation using the phonenumbers library, ensuring SMS campaigns only target valid UAE mobile numbers (+971).
Integrated Twilio API for automated SMS outreach campaigns, enabling real estate agents to engage with qualified leads immediately. Developed an audit schema tracking all data transformations, ensuring GDPR compliance and data lineage for regulatory requirements.
Technical Architecture
CSV Ingestion: Flask web interface accepts CSV uploads. Pandas reads and validates file structure before processing.
Data Validation Pipeline: phonenumbers library validates international format, filters for UAE mobile numbers, rejects non-UAE numbers.
Deduplication: SQL queries check existing leads by phone and email, preventing duplicate CRM entries.
Twilio Integration: Automated SMS campaigns for qualified leads with customizable templates.
Audit Trail: PostgreSQL schema logs every transformation for compliance and debugging.
Key Challenges & Solutions
International Phone Number Validation
Used Google's phonenumbers library to parse international formats. Implemented regex for UAE mobile patterns (+971 5X), reducing invalid SMS sends by 92%.
Pandas ETL Performance for Large Files
Optimized Pandas processing for 50K+ row CSVs using chunked reading (chunksize=1000), reducing memory usage by 70%.
Data Lineage and GDPR Compliance
Implemented comprehensive audit schema tracking source files, transformations, and delivery status, ensuring GDPR compliance.
Impact & Results
Reduced lead processing time from 4 hours/day to 10 minutes, a 95% efficiency gain
Improved SMS campaign effectiveness by 92% through phone validation
Enabled data-driven marketing with lead source attribution
GDPR-compliant audit trails for regulatory protection
Key Features
- CSV batch import and processing
- International phone number validation
- Automatic deduplication logic
- Twilio SMS campaign integration
- PostgreSQL relational database
- Flask web interface for data management
- Audit trail for all transformations
- Data validation with custom validators
- Schema migration scripts
- Error handling and rollback mechanisms
Technologies Used
Project Details
Client
Dubai Real Estate Agency - Freelance
Timeline
2023
Role
Python Backend Developer & Data Engineer
© 2026 Firas Jday. All rights reserved.