Lead Data Orchestration Engineer specialized in Apache Airflow and AWS MWAA

Project overview

Lead Data Orchestration Engineer specialized in Apache Airflow and AWS MWAA

The mission is structured around four axes:

Axis 1 — Architectural Optimization

Optimize and evolve existing orchestration mechanisms to enable scaling and reduce operational costs (incident resolution time, effort for adding new countries).

Axis 2 — Security and Resilience

Strengthen platform robustness against load and failures: end-to-end observability, secret rotation policy, flow isolation, automatic recovery capability.

Axis 3 — Study and Migration to a Target Infrastructure

The audit highlighted the structural limitations of AWS MWAA for our needs (scaling constraints, lack of configuration flexibility, costs). The Lead Data Orchestration Engineer will need to study alternatives (self-hosted Airflow on ECS/EKS, Astronomer, Dagster, Prefect, or another solution adapted to our processes) and lead the migration to the chosen infrastructure if relevant.

Axis 4 — Team Autonomy

Train and structure Dev and Ops teams so they can operate, diagnose, and evolve the platform autonomously in the long term.

Seniority: Minimum 5 years of experience with Apache Airflow, including significant experience in a managed environment (AWS MWAA or Google Cloud Composer).

Required technical skills:

Cross-functional skills:

Profile sought

Work environment

• Mastery of the Airflow execution model: scheduler, executor, pools, triggers, deferrable operators
• Experience in tuning and optimizing Airflow platforms at scale (200+ DAGs in production)
• AWS Expertise: MWAA, SQS, SNS, Secrets Manager, CloudWatch, IAM
• Knowledge of MWAA alternatives: Airflow on ECS/EKS, Astronomer, or modern orchestrators (Dagster, Prefect) — experience migrating from one orchestrator to another highly appreciated
• Experience with dynamic DAG Factories (generating DAGs from configurations) — ability to evolve and optimize an existing factory
• Resilience patterns: retry strategies, Dead Letter Queues, circuit breakers, flow isolation
• Observability: metrics, dashboards, alerting, incident routing by criticality
• Advanced Python (custom plugins, operators, sensors)
• Infrastructure as Code (Terraform) — knowledge desired
• Ability to design and enforce technical standards (code reviews, patterns, conventions)
• Experience in mentoring and skill transfer to mixed teams (Dev + Ops)
• Ability to lead a technical optimization program (prioritization, monitoring, reporting)
• Autonomy and proactivity

Project details

• Start: ASAP
• Duration: 6 mois renouvelable
• Remote: hybrid
• Location: France, FR
• Rate: 50-65 EUR/h
• Languages: French