Required skills
Languages: English
Job description
Project overview
This role is responsible for leading the ingestion workstream that reliably transfers all source data, starting with Oracle, into the bronze layer while providing a functional ingestion framework.
Main Responsibilities
Key Requirements
- Own the ingestion inventory: sources, tables, volumes, load patterns and priorities.
- Choose the ingestion pattern per source (Lakeflow Connect, Auto Loader, Azure Data Factory) and document the rationale.
- Design and build a metadata-driven ingestion framework, allowing new sources to be added through configuration rather than new code.
- Implement Change Data Capture (CDC) and incremental loads, accommodating schema changes, deletions, and late-arriving data.
- Design for idempotency and replay through checkpoints, watermarks, and controlled reprocessing.
- Establish monitoring, alerting, load SLAs, and data-quality gates at the bronze layer.
- Plan and execute cut-over operations for each source, including parallel runs, reconciliation, and sign-offs.
- Lead and mentor 2-3 data engineers, and ensure proper handover of the framework and runbooks.
- Extensive experience with Databricks and data ingestion frameworks.
- Proficiency in Oracle and related data management.
- Strong understanding of CDC and data load strategies.
- Experience in metadata-driven architecture design.
- Ability to lead and mentor engineering teams.
Nice to have
- Familiarity with Azure Data Factory.
- Knowledge of data quality monitoring tools.
- Experience with performance optimization of data ingestion processes.
Project details
ASAP
3-4 months
Copenhagen
- Start: flexible
- Duration: 4-6 months
- Remote: hybrid
- Location: Copenhagen, Denmark, DK
- Languages: English
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on portal.emagine.org