Project overview
The challenge
- Organizations migrating to new Google Workspace domains needed to transfer TB-scale Drive data without manual downloading/uploading.
- Google Drive API rate limits caused transfer failures when moving large file counts concurrently.
- No audit trail for compliance requirements (who transferred what files, when, and status).
- Incomplete transfers due to network errors left data in inconsistent states with no resumption capability.
Developed a robust Google Drive migration platform enabling seamless file and folder transfers between Google accounts with progress tracking, error recovery, and comprehensive audit trails.
Implemented p-queue for intelligent concurrency control, preventing API rate limit errors while maximizing transfer throughput. Winston logging provides detailed audit trails for compliance and debugging.
Integrated Firebase Admin for multi-user authentication and Google Cloud Secret Manager for secure credential storage. Deployed to GCP with automatic scaling for enterprise workloads.
Technical architecture
API Gateway: Express.js with Firebase authentication ensuring only authorized users can initiate migrations.
Concurrency Control: p-queue limits parallel Drive API calls to 5 concurrent workers, preventing rate limit errors while maintaining high throughput.
Transfer Workers: Each worker reads from source Drive, uploads to destination Drive, handling permissions and folder hierarchies.
Audit Logging: Winston logs every file transfer with timestamps, file IDs, sizes, and success/failure status for compliance.
Notification System: Handlebars templates generate progress emails sent via Nodemailer for long-running migrations.
Key challenges & solutions
Google Drive API Rate Limiting
Implemented p-queue with concurrency limit of 5 and exponential backoff. This prevented 403 rate limit errors while achieving 95% of theoretical maximum throughput.
Large-Scale Data Transfer Reliability
Built resumable transfer logic storing transfer state in Firestore. If migration paused/failed, system resumes from last successful file, preventing data duplication.
Winston Audit Trail for Compliance
Configured Winston with structured logging (JSON format), daily log rotation, and log shipping to GCP Cloud Logging for long-term audit retention.
Impact & results
Migrated 50TB+ of enterprise data across Google Workspaces
95% reduction in migration time vs manual download/upload
Zero data loss with resumable transfers and comprehensive logging
Full compliance audit trail with Winston structured logs
Key features
- Google Drive API file transfer
- p-queue concurrency management
- Winston structured logging
- Firebase Admin authentication
- OAuth 2.0 with Passport.js
- GCP Secret Manager integration
- Handlebars email templates
- Express-validator input validation
- Progress tracking & resumption
- Error recovery mechanisms