Lead Data Engineer

Project overview

Hello,

I am looking for a Lead Data Engineer:

Main missions:

Profile sought

Technical skills/environment:

Work environment

📍 Location: France

🏡 Modalities: Hybrid

🎢 Duration: Long-term visibility

👜 Status: Freelance or payroll management (company referenced in the PEPS union)

💰 Rate: 450-500 euros

• Optimize and evolve existing orchestration mechanisms to enable scaling
• Reduce operating costs and improve incident resolution times
• Strengthen the robustness and resilience of the Data platform
• Implement end-to-end observability, alerting, and advanced monitoring
• Define and apply secret management and rotation policies
• Design automatic recovery and flow isolation mechanisms
• Study alternatives to AWS MWAA (Airflow on ECS/EKS, Astronomer, Dagster, Prefect, etc.)
• Make architectural recommendations and manage migrations if necessary
• Evolve existing DAG Factories and industrialize orchestration processes
• Implement resilience best practices (DLQ, retry strategies, circuit breakers)
• Define and enforce technical standards, patterns, and development conventions
• Train and support Dev and Ops teams towards complete operational autonomy
• Ensure follow-up of improvement plans, reporting, and prioritization of technical projects
• Apache Airflow (5 years minimum)
• AWS MWAA
• AWS: SQS, SNS, Secrets Manager, CloudWatch, IAM
• Scheduler, Executor, Pools, Triggers, Deferrable Operators
• Optimization and tuning of large-scale Airflow platforms (200+ DAGs in production)
• Self-hosted Airflow on ECS/EKS
• Astronomer
• Dagster
• Prefect
• Dynamic DAG Factories
• Advanced Python (Operators, Sensors, custom Plugins)

Project details

• Start: ASAP
• Duration: 6 mois renouvelable
• Remote: hybrid
• Location: France, FR
• Rate: 100-500 EUR/h
• Languages: French