Data Engineer PySpark / Iceberg

Project overview

Profile Data Engineer PySpark / Iceberg – Optimization of electronic invoicing data processing

Context

Complex, high-volume processing. Electronic invoicing data is prepared up to the Silver layer. Apache Iceberg goes into production mid-December. The target volume is tens of billions of invoices. The model has nearly 30 tables, some with multiple records per invoice or invoice line.

Main mission

Data Engineer to support the team in code design, data flow software design, and transformation pipeline optimization:

Data modeling

Performance, optimization, and operation

Technical environment

Apache Spark, PySpark, Apache Iceberg, Hadoop, Hive, Python

Technical skills

Soft skills: explain design choices, document, transmit (code reviews)

• Provide a complementary perspective on platform usage and discuss approaches with the team
• Support the transition from full processing to reliable incremental processing
• Participate in experiments, benchmarks, and performance measurements
• Challenge the code structure and data flow design (business rules, usage, performance)
• Characterize volumes and their growth per table
• Explore maximum parallelization of processing
• Transition to incremental processing for complex cancellation and rectification cases: targeted cascading updates, data flow reconciliation, indicator accuracy
• Propose a review of business specifications if necessary
• Analyze existing processing (execution plans, production metrics) and prioritize optimizations
• Make PySpark modifications while preserving business rules, readability, and maintainability
• Utilize Iceberg: optimized MERGE, Hidden Partitioning
• Optimize the physical layout of the Bronze and Silver layers (Bucketing, Clustering, Z-Ordering)
• Ensure the quality, deduplication, and consistency of incoming data
• Measure gains before/after and transfer work to the team
• Apache Iceberg applied to code: MERGE optimization, Hidden Partitioning, ACID transactions
• Apache Spark / PySpark: job design, processor management, memory optimization, query tuning
• Python and Software Craftsmanship (Clean Code, refactoring)
• Bash and analytical SQL at the billion-row scale

Project details

• Start: flexible
• Remote: hybrid
• Location: 94120 Fontenay-sous-Bois, France, FR
• Rate: 72-72 EUR/h
• Languages: French