Senior Data Engineer - PySpark

Project overview

Context

As part of strengthening a team, we are looking for a Senior/Expert Data Engineer consultant capable of independently intervening in the design, industrialization, and operation of Data products and pipelines.

The mission takes place in an AWS Lakehouse environment, with challenges related to data quality, performance, processing reliability, and RUN management.

Missions

The consultant will notably be involved in the following activities:

Deliverables

The deliverables for the assignment may include:

Skills

Key Skills

Mandatory:

Important:

Interesting:

• Design, develop, and optimize data pipelines.
• Develop and industrialize processing with Spark/PySpark, Python, SQL, and dbt.
• Operate data in Apache Iceberg format within a Lakehouse / Medallion architecture.
• Orchestrate processing with Airflow.
• Deploy and maintain processing on AWS and contribute to Infrastructure as Code with Terraform.
• Implement Data Quality controls, tests, and observability mechanisms.
• Participate in RUN, incident analysis, and root cause treatment.
• Contribute to documentation, Data Lineage, Data Contracts, and Data Catalog.
• Work with Product Owners and business stakeholders to translate Marketing needs into robust and maintainable Data solutions.
• Developed, tested, and industrialized Data pipelines.
• Spark transformations and dbt models.
• Airflow workflows.
• Terraform infrastructure required for processing.
• Automated tests and Data Quality controls.
• Monitoring and alerting associated with processing.
• Technical and functional documentation.
• Documentation of business models and rules.
• Contribution to Data Contracts, lineage, and metadata.
• Incident analysis and resolution.
• Remediation plans for recurring incidents or problems.
• Contribution to the team's Data Engineering standards and best practices.
• 8 years minimum experience in Data Engineering
• Advanced mastery of Big Data and Lakehouse architectures
• Real expertise in Spark, DBT, Iceberg, and AWS services.
• Mastery of AWS services (Glue, EMR, S3, ...)
• Knowledge of Trino/Starburst, Datadog
• Strong analytical ability and understanding of business challenges.
• Rigor, autonomy, and a culture of continuous improvement.
• Knowledge of Marketing business challenges will be appreciated.
• SPARK Development
• AWS
• Python
• Big Data
• Data modeling tools
• Data collection
• DBT
• Starburst
• Data Catalog

Project details

• Start: flexible
• Remote: hybrid
• Location: Nantes, France, FR
• Languages: French