Project overview
The challenge
- Retail source data (products, stores, costs, sales, competitor prices) arrives as files in Azure, while the pricing and analytics stack runs on GCP.
- Pricing results computed in BigQuery must reach the PostgreSQL database that the application edits.
- Each retail client brings its own data grain and formats, so ingestion has to stay client-specific without forking the core platform.
Mercio is a pricing platform for retail chains: it turns product, store, cost and competitor data into explainable price recommendations. I worked on it as a full-stack contractor from June 2025 to January 2026, during the MVP rebuild of the pricing product.
On the data side, the Intermarché pipeline runs as six Python 3.12 Cloud Functions: one copies new source files from Azure to Cloud Storage, a storage event loads them into BigQuery, Dataform builds the intermediate, analysis and pricing tables, and a bridge writes the results back to the PostgreSQL edition database. A tagged CircleCI release deploys the functions, then runs Dataform and waits for it to finish.
The application side is a Gradle composite build of Java 17/Spring Boot services, a React/TypeScript front end, Keycloak for identity and Liquibase for schema migrations, packaged with Helm for GKE. A maintenance function resets stuck GKE pods.
Technical architecture
Ingestion: Python 3.12 Cloud Functions copy new Azure files to Cloud Storage and load them into BigQuery on storage events.
Transformation: Dataform builds the intermediate, analysis and pricing tables.
Sync: a bridge function writes BigQuery results to PostgreSQL by insert, upsert or truncate.
Application: Java 17/Spring Boot services and a React front end, with Keycloak and Liquibase, deployed with Helm on GKE.
Release: a tagged CircleCI release deploys the functions, then runs Dataform and waits for the result.
Key challenges & solutions
Client-specific ingestion on a shared platform
Intermarché ingestion and Dataform definitions live in their own repository, so the core pricing platform stays generic across retail clients.
Waiting for Dataform in the release
The release step polls the Dataform workflow until it succeeds or fails, instead of firing it and moving on.
Impact & results
Pricing data from Azure sources made available in BigQuery and PostgreSQL for the pricing application
Dataform transformations versioned in Git and run as part of each release
Stuck GKE pods reset by a maintenance function instead of by hand
Key features
- Azure → Cloud Storage copy of new source files (Cloud Functions, Python 3.12)
- Storage-event loading into BigQuery
- Dataform SQL transformations: intermediate, analysis and pricing tables
- BigQuery → PostgreSQL sync with insert, upsert or truncate modes
- Tagged CircleCI release that deploys the functions and waits for the Dataform run
- Maintenance function that resets stuck GKE pods
- Java 17 / Spring Boot services packaged with Helm for GKE
