Required skills
- PySpark
- Apache Iceberg
- Data Engineering
- Data Modeling
- Performance Tuning
- SQL
- Python
- Apache Spark
- ACID Transactions
- Clean Code
- Refactoring
- Bash
Languages: French
Job description
Seeking a Data Engineer with PySpark and Apache Iceberg experience to optimize a high-volume electronic invoicing data processing pipeline. The role involves improving data transformation pipelines, transitioning to incremental processing, and performance tuning.
Project overview
Profile Data Engineer PySpark / Iceberg – Optimization of electronic invoicing data processing
Context
Complex, high-volume processing. Electronic invoicing data is prepared up to the Silver layer. Apache Iceberg goes into production mid-December. The target volume is tens of billions of invoices. The model has nearly 30 tables, some with multiple records per invoice or invoice line.
Main mission
Data Engineer to support the team in code design, data flow software design, and transformation pipeline optimization:
Data modeling
Performance, optimization, and operation
Technical environment
Apache Spark, PySpark, Apache Iceberg, Hadoop, Hive, Python
Technical skills
Soft skills: explain design choices, document, transmit (code reviews)
- Provide a complementary perspective on platform usage and discuss approaches with the team
- Support the transition from full processing to reliable incremental processing
- Participate in experiments, benchmarks, and performance measurements
- Challenge the code structure and data flow design (business rules, usage, performance)
- Characterize volumes and their growth per table
- Explore maximum parallelization of processing
- Transition to incremental processing for complex cancellation and rectification cases: targeted cascading updates, data flow reconciliation, indicator accuracy
- Propose a review of business specifications if necessary
- Analyze existing processing (execution plans, production metrics) and prioritize optimizations
- Make PySpark modifications while preserving business rules, readability, and maintainability
- Utilize Iceberg: optimized MERGE, Hidden Partitioning
- Optimize the physical layout of the Bronze and Silver layers (Bucketing, Clustering, Z-Ordering)
- Ensure the quality, deduplication, and consistency of incoming data
- Measure gains before/after and transfer work to the team
- Apache Iceberg applied to code: MERGE optimization, Hidden Partitioning, ACID transactions
- Apache Spark / PySpark: job design, processor management, memory optimization, query tuning
- Python and Software Craftsmanship (Clean Code, refactoring)
- Bash and analytical SQL at the billion-row scale
Project details
- Start: flexible
- Remote: hybrid
- Location: 94120 Fontenay-sous-Bois, France, FR
- Rate: 72-72 EUR/h
- Languages: French
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on collective.work