Geforderte Skills
- Apache Spark
- PySpark
- Spark SQL
- Structured Streaming
- MongoDB
- Apache Iceberg
- Data Lakehouse
- Python
- Jenkins
- Prometheus
- Grafana
- Scrum
Sprachen: Polish
Jobbeschreibung
We are looking for an experienced specialist for the Data Platform team to build and maintain data layers in a Data Lakehouse architecture. The role involves remote work with occasional office visits.
Project overview
We are looking for an experienced specialist for the Data Platform team for one of our clients in the financial sector. The project focuses on building and maintaining data layers in a Data Lakehouse architecture. This includes the full processing cycle, from raw datasets to presentation layers (curated, silver, gold). The solutions are based on a modern technology stack, including Apache Iceberg and MongoDB in the ODS (Operational Data Store) layer.
Remote work, with one day per month visit to the office in Wrocław.
Your responsibilities
What we expect
- Designing and implementing data reconciliation solutions in the MongoDB-based ODS layer.
- Synchronizing data from source systems.
- Building and maintaining data layers in a Data Lakehouse architecture (Delta Lake, Apache Iceberg, Hudi).
- Managing versioned tables and optimizing writes (upserts, merge) in Apache Iceberg.
- Automating the build, testing, and deployment processes of Spark applications using Jenkins.
- Monitoring solutions using Prometheus and Grafana tools.
- Minimum 4 years of practical experience with Apache Spark in production environments.
- Proficient knowledge of PySpark, Spark SQL, DataFrames, and Structured Streaming.
- Experience working with MongoDB, including reading and writing data via Spark Connector, document modeling, and indexing.
- Practical knowledge of Data Lakehouse architecture and tools such as Apache Iceberg.
- Ability to create CI/CD processes using Jenkins.
- Proficiency in Python programming.
- Knowledge of system monitoring issues (Prometheus, Grafana).
- Experience working with agile methodologies (Scrum or Kanban) and knowledge of Jira and Confluence.
Project details
- Start: flexible
- Remote: hybrid
- Location: Wrocław, -, PL
- Rate: 125-145 EUR/h
- Languages: Polish
Indexiert von der 9am-Meta-Suche. Du bist Job-Anbieter? Listing beanspruchen oder verwalten
Gefunden auf justjoin.it