Consultant Data LakeHouse Architect (Python, Cloudera, Iceberg)

Project overview

We are looking for a Consultant Data LakeHouse Architect (Python, Cloudera, Iceberg) for a long-term mission in Vélizy (78).

Within the Data Factory, you will be involved in Data Architecture design, data engineering, and data set integration issues in a Big Data & Cloud environment.

As a Big Data & Cloud Data Engineer, with a Data Architecture dimension, you will contribute to the design, development, and evolution of the Data LakeHouse (DLH) architecture, as well as the preparation and integration of data required for the PRM project.

Your main responsibilities

In this context, your missions will include:

Profile sought

Throughout the assignment, the following deliverables will be expected:

Work environment

We are looking for an experienced profile in Big Data & Cloud Data Engineering, with solid technical expertise and the ability to work on data architecture and integration issues.

Essential technical skills

Experience and education

Complementary skills

• Data Architecture: participate in the design, implementation, and evolution of the Data LakeHouse architecture on a hybrid Data platform based on Cloudera.
• Data Engineering: design, develop, and optimize ETL pipelines in Python and PySpark to ensure data transformation, preparation, and integration.
• Data Management: implement and operate Apache Iceberg tables to ensure efficient and structured management of data sets.
• Migration and Interoperability: evolve existing data systems to ensure their compatibility with the PRM project and facilitate their migration.
• Data Modeling: define and maintain reusable, consistent, and optimized data models, prioritizing non-redundancy and uniqueness of definitions.
• Data Quality and Governance: ensure the quality, consistency, and reliability of integrated data, in compliance with data engineering standards and best practices.
• Security and Compliance: integrate security requirements, regulatory compliance (particularly GDPR and regulations applicable to data and artificial intelligence), as well as intellectual property issues, including for data from third-party providers.
• Cross-functional Collaboration: work closely with application and technical architects, Data Scientists, Data Analysts, and Legal and Research teams to validate models and data content produced.
• Development of ETL jobs for data retrieval, transformation, and preparation for integration into the Data LakeHouse (DLH).
• Development of ETL jobs for data preparation and export to the PRM project.
• Associated technical documentation, including:
• Data models;
• Data schemas;
• Integration flow documentation;
• Technical elements necessary for the maintenance and evolution of processing.
• Excellent command of Python and PySpark.
• Good knowledge of Apache Iceberg.
• Mastery of hybrid Data platform architectures, ideally on the Cloudera environment.
• Good understanding of Data LakeHouse architectures.
• Proven experience in designing, developing, and optimizing ETL pipelines in a Big Data & Cloud environment.
• Mastery of large-scale data transformation, preparation, and integration processes.
• Higher education degree (Bac+5 level
• engineering school, university, or equivalent), specializing in IT, Data, or information systems.
• Minimum 4 to 5 years of experience in Big Data & Cloud Data Engineering, ideally on projects involving the construction or evolution of Data platforms.
• Knowledge of the SAP environment would be an asset.
• Sensitivity to data governance, quality, and security issues is expected.
• Ability to work in an international and multicultural environment.
• Professional English required, both spoken and written.

Project details

• Start: ASAP
• Duration: 6 mois renouvelable
• Remote: hybrid
• Location: Vélizy-Villacoublay, Île-de-France, FR
• Languages: French