Lead Data Orchestration Engineer

Project overview

Mission Context

As part of strengthening its Data Platform team, our client is looking for a Lead Data Orchestration Engineer, an expert in Apache Airflow and AWS MWAA.

The current platform orchestrates banking data pipelines in 9 European countries. After an initial stabilization phase, the objective is now to optimize its architecture, strengthen its resilience, and study a potential migration to more suitable infrastructure.

Missions and Responsibilities

Essential Technical Skills

Appreciated Skills

Essential Skills / Qualities:

Apache Airflow, AWS MWAA or Google Cloud Composer, Data Orchestration, Lead Data Engineer, Python, DAG Factory, AWS, ECS, EKS, Astronomer, Dagster, Prefect, DataOps, Terraform, CI/CD, CloudWatch

Profile Sought

• Airflow Architecture Optimization
• Evolve and optimize the existing DAG Factory.
• Migrate inter-DAG dependencies to native Airflow mechanisms (Datasets).
• Improve platform performance and scalability.
• Automate DAG validation in CI/CD pipelines.
• Security and Resilience
• Enhance observability: monitoring, dashboards, alerting.
• Implement a secret rotation policy.
• Optimize capacity planning, scaling, and resource management.
• Improve network isolation and automatic recovery mechanisms.
• Formalize incident management procedures (runbooks).
• Infrastructure Study and Migration
• Benchmark alternatives to AWS MWAA (ECS/EKS, Astronomer, Dagster, Prefect).
• Evaluate associated costs, performance, and risks.
• Define a target architecture and a migration plan.
• Lead POCs and the progressive migration if a new solution is chosen.
• Ensure service continuity during the transition.
• Support and Skill Development
• Train and support Dev and Ops teams (approximately 10 people).
• Define and enforce technical standards.
• Structure a Data Platform organization that promotes team autonomy.
• Ensure knowledge transfer and mentoring.
• Apache Airflow: advanced expertise (scheduler, executor, pools, triggers, deferrable operators, Datasets).
• AWS: MWAA, SQS, SNS, Secrets Manager, CloudWatch, IAM.
• Advanced Python: development of custom plugins, operators, and sensors.
• Airflow Optimization: experience with large-scale platforms (200+ DAGs in production).
• DAG Factory: design and optimization of dynamic DAG generation.
• Resilience and Observability: monitoring, alerting, retry strategies, Dead Letter Queues, circuit breakers.
• Cloud Architecture: ability to evaluate and design scalable orchestration solutions.
• Experience with an orchestrator or Airflow infrastructure migration.
• Knowledge of Airflow on ECS/EKS, Astronomer, Dagster, or Prefect.
• Proficiency in Terraform (Infrastructure as Code).
• Experience in a banking, international, or multi-country environment.
• Minimum 5 years of experience with Apache Airflow, with significant experience on AWS MWAA or Google Cloud Composer.
• Proven experience in architecture, optimization, and operation of Data orchestration platforms.
• Ability to lead technical transformations and define architectural directions.
• Technical leadership, autonomy, and proactivity.
• Excellent communication, mentoring, and Dev/Ops team coordination skills.

Project details

• Start: 25/10/2026
• Duration: 3 mois renouvelable
• Remote: hybrid
• Location: Sochaux, Bourgogne-Franche-Comté, FR
• Rate: 81-81 EUR/h
• Languages: French