Senior Data Engineer

Project overview

We are looking for a Senior Data Engineer for the design and implementation of a production-ready Databricks monitoring solution for cross-border electricity exchanges using existing data sources to support automated analysis, quality checks, KPI calculation, dashboards, and alerting.

Further Framework Data

Type of Assignment:

Utilization: 50%

Industry: Other

Responsibilities

Must-Have Requirements

Nice-to-Have Requirements

Let’s power the future together

From business case to implementation: As a leading consulting firm for strategic transformations, we are a trusted partner for our clients - and for our employees. Responsible, high-performing, and always with a focus on people. #WeAreWavestone

With our 360° portfolio of consulting services, we combine first-class industry expertise with a broad spectrum of cross-industry competencies, work interdisciplinarily, and think outside the box. We can thus offer our partner companies and freelancers comprehensive perspectives within our own projects and support them as a long-term framework agreement partner in filling project vacancies – promptly and directly.

We look forward to hearing from you!

Your direct contact at Wavestone

Elma Abdagic

Phone: *

Email: *@*.eu

• Analyze available data sources, data products, technical interfaces, existing data structures, and data flows and adapt them for functional and technical monitoring
• Design a target architecture for the monitoring solution using Databricks and existing data sources
• Define KPIs, thresholds, and monitoring logic to identify unintended power flows
• Implement automated KPI calculations and validation mechanisms to detect anomalies, data quality issues, transmission errors, and missing or faulty data deliveries
• Develop monitoring dashboards, analysis results, and associated artifacts for operational and analytical use
• Implement monitoring and alarm systems for deviations, data availability issues, and error conditions
• Document data models, processing logic, technical interfaces, and operational procedures
• Conduct tests and create test documentation, acceptance test certificates, and production deployment artifacts; hand over a production-ready, traceable, and maintainable solution including source code, configurations, and documentation
• At least 3 years of relevant practical experience in Data Engineering, including experience in developing Databricks-based data solutions
• Very good knowledge of Apache Spark, Spark SQL, and distributed data processing concepts
• Very good Python programming skills
• Extensive SQL experience, including optimization of complex queries
• Proven knowledge of modern Data Lake and Lakehouse architectures
• Proven experience in processing large amounts of data, including second-accurate time series, and in optimizing the performance of data processing workflows
• Demonstrable experience in developing monitoring and reporting artifacts, dashboards, and analytics applications, including dashboard design
• German language skills C1 level
• Experience with operational monitoring and alerting concepts in Databricks environments
• Experience in documenting data models, technical interfaces, processing logic, and operational procedures
• Experience in conducting test activities and creating acceptance documentation and production deployment deliverables for data solutions
• Experience in the energy sector or with time series data from energy system processes
• Experience with time conventions, including UTC, local time, and daylight saving time changes

Project details

• Start: flexible
• Location: DE
• Languages: German