Data Engineer AWS / Scala Spark

Project overview

We are looking for a senior Data Engineer with strong expertise in AWS, Scala, and Spark environments, ideally gained in contexts with high data volumes.

Within a central Data department, the Data Platform team designs, develops, and maintains a centralized, secure, scalable, and high-performing data platform to support cross-functional data uses for various business departments. It notably operates a data lake hosted on AWS, a data processing infrastructure based on Scala Spark, as well as an ecosystem for orchestrating and industrializing jobs via Airflow. The team also ensures the maintenance and evolution of a Dataiku platform.

The mission takes place in an agile environment, interacting with product, data, BI, analytics, data science, and business teams.

The consultant will be involved in two main areas.

Operational Maintenance

Build & Evolution of Data Pipelines

Profile Sought

The profile should also have a strong functional appetite to understand business challenges, challenge needs, and communicate effectively with various stakeholders.

Minimum 5 years of experience

Work Environment

LOMEGARD is a consulting firm specializing in IT professions, offering 4 cross-functional services: PROJECT, SOFTWARE, DATA-IA, and INFRASTRUCTURE.

We support our clients in their digital transformation and innovation projects by creating a more respectful work environment for the planet.

Key Figures:

Visit our website to learn more.

May LOMEGARD be with you!

• Monitor the availability of data and platform services
• Analyze production incidents and anomalies
• Respond to ad hoc data-related requests
• Develop necessary fixes
• Contribute to the quality, reliability, and industrial operation of the platform
• Communicate with users and stakeholders
• Design and develop robust and performant data pipelines
• Implement data acquisition, transformation, and export flows
• Perform distributed processing in Scala Spark
• Participate in data modeling
• Develop unit and integration tests
• Manage scheduling and dependencies between processes
• Participate in code reviews and adherence to standards
• Perform deployments on different environments
• Maintain technical and functional documentation
• Advanced expertise in Scala Spark
• Solid experience on AWS Data & Analytics
• Good command of Python and SQL
• Experience with Airflow
• Knowledge of Kafka / MSK
• Practice with GitLab, CI/CD, Maven
• Knowledge of Terraform
• Ability to work on performance, security, cost, and scalability issues
• Understanding of data quality, governance, and reliability challenges
• Experience in an agile environment
• Founded in September 2023
• Already nearly 40 LOMEGARDIANS
• 4 million turnover by 2025
• 12 key accounts

Project details

• Start: ASAP
• Duration: 1 an renouvelable
• Remote: hybrid
• Location: Paris, France, FR
• Languages: French