Senior/Expert Data Engineer Consultant

Project overview

Context

As part of strengthening the Marketing Data team within the Data Fabric unit, we are looking for a Senior/Expert Data Engineer Consultant capable of independently intervening in the design, industrialization, and operation of Data products and pipelines.

The mission takes place in an AWS Lakehouse environment, with challenges related to quality, performance, processing reliability, and RUN management.

Mission Description

The consultant will notably be involved in the following activities:

Deliverables

The deliverables of the service may include:

• Design, develop, and optimize data pipelines
• Develop and industrialize processing with Spark/PySpark, Python, SQL, and dbt
• Operate data in Apache Iceberg format within a Lakehouse / Medallion architecture
• Orchestrate processing with Airflow
• Deploy and maintain processing on AWS and contribute to Infrastructure as Code with Terraform
• Implement Data Quality controls, tests, and observability mechanisms
• Participate in RUN, incident analysis, and root cause treatment
• Contribute to documentation, Data Lineage, Data Contracts, and Data Catalog
• Work with Product Owners and business stakeholders to translate Marketing needs into robust and maintainable Data solutions
• Developed, tested, and industrialized Data pipelines
• Spark transformations and dbt models
• Airflow workflows
• Terraform infrastructure required for processing
• Automated tests and Data Quality controls
• Monitoring and alerting associated with processing
• Technical and functional documentation
• Documentation of business models and rules
• Contribution to Data Contracts, lineage, and metadata
• Incident analysis and resolution
• Remediation plans for recurring incidents or problems
• Contribution to the team's Data Engineering standards and best practices

Project details

• Start: flexible
• Remote: hybrid
• Location: Nantes, France, FR
• Languages: French