Required skills
- Databricks
- PySpark
- Spark SQL
- Delta Lake
- Unity Catalog
- Databricks Workflows
- Azure Data Factory
- Apache Airflow
- Terraform
- CI/CD
- DevOps
Languages: French
Job description
We are looking for an experienced Data Engineer specialized in the Databricks ecosystem to design, develop, and industrialize high-performance data pipelines in a Cloud environment.
Project overview
Mission Context
We are looking for an experienced Data Engineer specialized in the Databricks ecosystem, capable of designing, developing, and industrializing high-performance data pipelines in a Cloud environment.
The consultant will work on high-value Data projects, with challenges related to performance, governance, quality, and data reliability.
Responsibilities
- Design and develop complex data pipelines in PySpark and Spark SQL, in batch and streaming modes.
- Implement and maintain a Medallion architecture (Bronze / Silver / Gold) based on Delta Lake.
- Orchestrate and monitor processing via Databricks Workflows, Azure Data Factory, or Airflow.
- Optimize Spark processing performance and costs: partitioning, Z-Order, cluster management, and Photon.
- Implement data governance with Unity Catalog: access management, data lineage, and data quality.
- Industrialize developments and deployments through CI/CD pipelines and DevOps practices.
- Participate in the implementation of Infrastructure as Code with Terraform and the management of Databricks Asset Bundles.
- Collaborate with Data Analysts and Data Scientists teams to provide reliable, documented, and usable datasets.
Project details
- Start: flexible
- Remote: hybrid
- Location: Paris, France, FR
- Rate: 56-56 EUR/h
- Languages: French
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on collective.work