Data Engineer (H/F)

Project overview

The mission involves working on Data Engineering projects and developing Data platforms, with the objective of designing, developing, industrializing, and optimizing data processing pipelines in an AWS and Databricks Cloud environment.

The consultant will be responsible for:

Profile Sought

An experienced to senior Data Engineer with solid experience in development and data processing, strong SQL expertise, and good command of Big Data and Cloud environments.

The candidate must be able to work autonomously on the development and optimization of Data pipelines, while mastering the challenges of performance, quality, industrialization, and reliability of processing.

Languages

Expected Technical Skills

SQL / SQL Optimization: Expert

Python: Experienced

PySpark: Advanced

Databricks: Experienced

DBT: Experienced

AWS: Advanced

Airflow / Astro Scheduler: Advanced

GitHub CI/CD: Experienced

GitHub Copilot / AI-Assisted Development: Experienced

Agile Methodology: Experienced

Work Environment

The consultant will join a team working in a modern technological environment, with significant challenges around Data, industrialization of processing, and performance optimization.

• Data Development: designing and developing data pipelines in Python and PySpark to collect, transform, and process large volumes of data
• SQL Expertise: writing complex SQL queries, ensuring data modeling and transformation, and optimizing the performance of processing and queries
• Databricks & DBT: developing and maintaining data processing, structuring transformations, and ensuring data reliability and quality
• Workflow Orchestration: implementing and managing the scheduling of pipelines and processing with Airflow / Astro Scheduler
• Industrialization & CI/CD: implementing continuous deployment and continuous integration processes with GitHub CI/CD, ensuring code quality, maintainability, and reliability
• AWS Cloud Environment: working on a Cloud Data architecture and utilizing the necessary AWS services for data storage, processing, and management
• AI-Assisted Development: using GitHub Copilot to accelerate development, improve productivity, and optimize code quality
• Agile Methodology: participating in Agile ceremonies, collaborating with various technical and functional teams, and contributing to the continuous improvement of solutions
• French: bilingual
• English: intermediate
• Start Date: asap
• Duration: long-term mission
• Location: Lille
• Remote Work: 2 days per week

Project details

• Start: ASAP
• Duration: 1 an renouvelable
• Remote: hybrid
• Location: Lille, Hauts-de-France, FR
• Languages: French