Senior Data Engineer Consultant - Freelance

Project overview

Context

As part of strengthening the Marketing Data team, we are looking for a Senior/Expert Data Engineer consultant capable of independently intervening in the design, industrialization, and operation of Data products and pipelines.

The mission takes place in an AWS Lakehouse environment, with challenges related to quality, performance, processing reliability, and RUN management.

Missions

The consultant will notably intervene on the following activities:

Deliverables

The deliverables of the service may include:

Skills

Key Skills

Mandatory

Important

Interesting

• Design, develop, and optimize data pipelines.
• Develop and industrialize processing with Spark/PySpark, Python, SQL, and dbt.
• Operate data in Apache Iceberg format within a Lakehouse / Medallion architecture.
• Orchestrate processing with Airflow.
• Deploy and maintain processing on AWS and contribute to Infrastructure as Code with Terraform.
• Implement Data Quality controls, tests, and observability mechanisms.
• Participate in RUN, incident analysis, and root cause treatment.
• Contribute to documentation, Data Lineage, Data Contracts, and Data Catalog.
• Work with Product Owners and business stakeholders to translate Marketing needs into robust and maintainable Data solutions.
• Developed, tested, and industrialized Data pipelines.
• Spark transformations and dbt models.
• Airflow workflows.
• Terraform infrastructure necessary for processing.
• Automated tests and Data Quality controls.
• Monitoring and alerting associated with processing.
• Technical and functional documentation.
• Documentation of business models and rules.
• Contribution to Data Contracts, lineage, and metadata.
• Analysis and resolution of incidents.
• Remediation plans for recurring incidents or problems.
• Contribution to the team's Data Engineering standards and best practices.
• Minimum 8 years of experience in Data Engineering
• Advanced mastery of Big Data and Lakehouse architectures
• Real expertise in Spark, DBT, Iceberg, and AWS services.
• Mastery of AWS services (Glue, EMR, S3, ...)
• Knowledge of Trino/Starburst, Datadog
• Strong analytical ability and understanding of business challenges.
• Rigor, autonomy, and a culture of continuous improvement.
• Knowledge of Marketing business challenges will be appreciated.
• SPARK Development
• AWS
• Python
• Big Data
• Data modeling tools
• Data Collection
• DBT
• Starburst
• Data Catalog

Project details

• Start: flexible
• Remote: hybrid
• Location: Nantes, France, FR
• Rate: 70-70 EUR/h
• Languages: French