Data Engineer - Senior

Project overview

The Data Engineer creates the data infrastructure in a simple and efficient matter by design, build and integrate data from various sources.

The Data Engineer builds data pipelines from existing data sources and other (on-prem) systems into the Big Data platform; Creates consistent e2e BI solutions with a data lifecycle starting with collecting the needed data until visualizing them; Creates AI/ML models in BigData use cases and helping business to leverage the results; User story creation with good understanding of business processes, product and customer journey; Rates user stories for customer value and to challenge the product owner on prioritization of user stories (e.g., stability versus functionality); User story refinement: Is able to translate user stories into technical stories; Development: Able to design, develop and optimize data storage and data processing solutions and proficient in in one or more program languages like python and SQL to ensure sufficient quality level ,including test criteria; Deployment: Is able to create and execute script based deployments, including rollbacks.

Is able to maintain/setup automated deployment using the VF standard toolset; Operations: Is able to setup automated monitoring and proactive alerting, is able to provide parameters to size the underlying application infrastructure. The Data Kong Team is responsible for building and maintaining the Asset Analytics Wind data pipelines on Databricks, delivering trusted and scalable data products up to the Silver layer and enabling Value Delivery Teams through platform support, infrastructure management, governance, and technical expertise.

We are looking for a highly experienced, structured, and self-driven Senior Data Engineer / Databricks Platform Engineer who can independently own complex technical challenges, support the wider team, and contribute across both data engineering and platform engineering domains. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake. Build and support Bronze and Silver layer data products following Medallion architecture principles. Develop and optimize batch and streaming ingestion pipelines.

Monitor, troubleshoot, and improve Databricks performance, Spark jobs. Support Value Delivery Teams with platform onboarding, troubleshooting, best practices, and operational support. Implement and maintain Databricks and supporting Azure infrastructure using Terraform. Manage security, access control, and governance using RBAC, Unity Catalog, Service Principals, and Managed Identities. Develop and maintain CI/CD pipelines for automated deployment of data and infrastructure components. Ensure platform reliability, scalability, cost optimization, and operational excellence.

Required Skills Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake. Ability to troubleshoot complex production issues across, data, infrastructure, identity and platform layers. Good knowledge of distributed data processing, including Spark optimization, partitioning, joins, AQE, data skew handling, and Delta optimization. Experience with Structured Streaming and event-driven architectures. Knowledge of Databricks Asset Bundles. Experience in Terraform and Infrastructure-as-Code. Experience with Azure DevOps/GitHub, CI/CD pipelines, and deployment automation.

Good understanding of RBAC, Unity Catalog, Azure Entra ID, Key Vault, and data governance. Experience working with Azure Data Lake Storage (ADLS Gen2) and cloud-native data platforms. Nice to Have Experience with Databricks Declarative Pipelines (Delta Live Tables), including Streaming Tables, Materialized Views, data quality expectations, and pipeline orchestration. Experience working with OT, IoT, telemetry, or high-volume time-series data HBO/WO English is the main language we use for communication since the members of our team have different nationalities.

We expect the candidates to have a proven trackrecord in the data engineering area The team is present in the Amsterdam office on Tuesdays and Thursdays and we expect the candidate to also be avaiable in Amsterdam on those days. Experience in the energysector is a pre

Project details

• Start: 2027-01-11T23:00:00
• Remote: hybrid
• Location: Amsterdam Noord-Holland, NL
• Languages: English