Required skills
- Apache Spark
- PySpark
- Spark SQL
- Structured Streaming
- MongoDB
- Apache Iceberg
- Jenkins
- CI/CD
- Prometheus
- Grafana
- Scrum
- Kanban
Languages: Polish
Job description
We are looking for an experienced data engineer to join the Data Platform team for a financial sector client. The project involves developing an advanced data architecture, focusing on data reconciliation solutions in the ODS layer and building a modern Data Lakehouse environment.
Project overview
We are looking for an experienced data engineer to join the Data Platform team for one of our financial sector clients. The project involves developing an advanced data architecture. You will focus on designing data reconciliation solutions in the ODS layer and building a modern Data Lakehouse environment. Remote work, with a visit to the office in Wrocław once a month.
Project details
- Start: ASAP
- Remote: hybrid
- Location: Remote, Wrocław, PL
- Rate: 140-175 EUR/h
- Languages: Polish
Source notes
If, after reading this announcement, you feel that this is a role where you can truly build something – it is very likely that we are looking for you.
Ideamotive
What is Ideamotive?
Our requirements
- remote work option
- flexible working hours
- integration meetings
- An agile company providing tech team extension IT services for corporate clients (mainly in Poland).
- Our company has been recognized by Forbes for 3 consecutive years as one of the fastest-growing IT organizations in Poland and was included in the Financial Times 1000 list.
- The delivery team consists of 60 people, the core business team of 12 people, and we are aiming for growth, so a "can do" attitude and a focus on action are essential here.
- We have a very cohesive, engaged team, we enjoy what we do.
- If such an atmosphere aligns with your expectations, apply! Let's see what story we can write together.
- Design and implementation of data reconciliation solutions in the ODS layer based on MongoDB.
- Synchronization of data from source systems.
- Building and maintaining data layers in a Data Lakehouse architecture (Delta Lake, Apache Iceberg, Hudi).
- Processing data from the raw layer to presentation layers (curated, silver, gold).
- Minimum 4 years of commercial experience working with Apache Spark in production environments.
- Practical knowledge of PySpark, Spark-SQL, DataFrames, and Structured Streaming.
- Experience working with MongoDB, including document modeling, indexing, and using Spark Connector.
- Practical knowledge of Apache Iceberg, managing versioned tables, and optimizing writes.
- Knowledge of CI/CD tools, particularly Jenkins for deployment automation.
- Experience with monitoring tools such as Prometheus and Grafana.
- Knowledge of agile methodologies (Scrum, Kanban) and tools Jira and Confluence.
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on nofluffjobs.com