Skip to content
Job boardOpen

Data Engineer (Spark)

Upvanta sp. z o.o. · Wrocław, -, Poland · Remote · SENIOR

  • Wrocław, -, Poland · Remote
  • Polish

Indexed (2 days ago) · refreshed constantly

Budget
zł 150–163 / h
Start
flexible
Work mode
Remote
Contract
Freelance / B2B
Seniority
SENIOR
Reference
upvanta-sp-z-o-o--data-engineer-spark--wroclaw-data-af65b071

Required skills

Nice to have:Delta LakeApache HudiDockerKubernetesSpark OperatorOpenTelemetryDynatraceAgileJiraConfluence

Languages: Polish

Job description

Project overview

We are looking for an experienced Data Engineer/Spark / Data Lakehouse Developer to join the Data Platform team and participate in the development of modern data processing and management solutions.

This role will be responsible for designing and implementing data reconciliation mechanisms in the ODS (Operational Data Store) layer, based on MongoDB, as well as synchronizing data with source systems. An important part of the project will also be the construction and development of the Data Lakehouse architecture, covering data processing from the raw layer to the presentation layers (Raw, Bronze, Silver, Gold).

Responsibilities

Data reconciliation and quality

Integration with MongoDB

Data Lakehouse architecture development

Performance optimization

Monitoring and observability

DevOps and operational support

Team collaboration

  • Designing and implementing data reconciliation processes using Apache Spark, PySpark, and Spark SQL.
  • Comparing data stored in MongoDB with data in source systems.
  • Creating data validation and quality rules, and detecting duplicates, missing data, and anomalies.
  • Generating reports and alerts regarding data inconsistencies.
  • Integrating Apache Spark with MongoDB using the MongoDB Spark Connector.
  • Implementing data read and write processes.
  • Optimizing read operations, including partitioning, predicate push-down, and schema-aware reads.
  • Designing and developing Raw, Bronze, Silver, and Gold data layers.
  • Implementing solutions based on Apache Iceberg, Delta Lake, or Apache Hudi.
  • Managing versioned tables and optimizing data write and update operations (upsert/merge).
  • Implementing CI/CD for table schemas, migrations, and versioning.
  • Tuning Apache Spark processes, including partitioning, caching, broadcast joins, and Adaptive Query Execution.
  • Optimizing queries and data processing costs in the Data Lakehouse environment.
  • Configuring monitoring and metrics using Prometheus and Grafana.
  • Integrating logging and distributed tracing using OpenTelemetry or Dynatrace.
  • Defining SLAs/SLOs and alerts regarding data freshness and task completion accuracy.
  • Automating Spark application deployments using Docker, Kubernetes, and Spark Operator.
  • Diagnosing issues related to data quality, delays, and inconsistencies.
  • Creating and maintaining technical documentation, runbooks, and data flow diagrams.
  • Collaborating with Data Engineers, Business Analysts, testers, and the DevOps team.
  • Participating in work conducted using Scrum/Kanban methodologies.
  • Conducting code reviews and technical workshops.

Requirements

  • Minimum 6 years of commercial experience working with Apache Spark and production environments. Banking experience is a plus.
  • Very good knowledge of PySpark, Spark SQL, DataFrames, and Structured Streaming.
  • Practical experience in integrating Apache Spark with MongoDB using the MongoDB Spark Connector.
  • Knowledge of document modeling and indexing in MongoDB.
  • Experience with Apache Iceberg, particularly in table management, data versioning, and upsert/merge operations.
  • Practical knowledge of Python.
  • Experience in building and maintaining CI/CD processes using Jenkins.
  • Knowledge of Prometheus and Grafana tools.
  • Experience working with Agile methodologies (Scrum/Kanban).
  • Knowledge of Jira and Confluence.

Nice to have

Join the project!

If you have experience in building efficient Big Data solutions, know Apache Spark well, and want to develop a modern Data Lakehouse architecture, we invite you to apply!

  • Experience with Delta Lake or Apache Hudi.
  • Knowledge of Docker and Kubernetes and Spark Operator.
  • Practical knowledge of OpenTelemetry or Dynatrace.
  • Experience in designing monitoring solutions for batch and streaming processes.
  • Knowledge of Data Quality, Data Observability, and data lifecycle management concepts.

Project details

  • Start: flexible
  • Remote: remote
  • Location: Wrocław, -, PL
  • Rate: 150-163 EUR/h
  • Languages: Polish

Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing

Found on justjoin.it

  1. Job boardNew

    AWS Data Engineer

    Confidential· India· On-site· Senior

    • AWS
    • Amazon S3
    • AWS Glue
    • +9

Browse all jobs
Sign up free to apply