Databricks Consultant / Engineer – Energy & Utilities | Freelance

Project overview

Project: Build & Further Development of a Databricks-based Data Platform

For one of our clients in the German energy and utilities sector, we are currently looking for an experienced Databricks Consultant / Engineer to support an ongoing data platform program.

As part of the modernization of the data landscape, various operational and energy industry data sources are being consolidated on a central Databricks Lakehouse Platform and made available for analytics, forecasting, and other data-driven use cases.

We are explicitly looking for a freelancer who has already built productive Databricks environments and brings both architectural understanding and hands-on experience in data engineering, not just a conceptual consultant.

Responsibilities

The combination of Lakehouse, Delta Lake, Spark, and Unity Catalog also corresponds to typical productive Databricks architectures; Databricks describes Bronze, Silver, and Gold layers for Energy & Resources, among other things, as well as provision via Unity Catalog.

• Build and further develop a Databricks Lakehouse architecture
• Develop scalable ETL/ELT data pipelines
• Process and transform large amounts of data with Apache Spark / PySpark
• Build and optimize data models based on Delta Lake
• Implement Medallion architectures (Bronze / Silver / Gold)
• Integrate various source systems into the central data platform
• Implement governance, authorization, and lineage concepts with Unity Catalog
• Performance optimization of existing Databricks workloads and pipelines
• Collaborate with Data Architects, Data Engineers, and Analytics and Business Departments
• Support the migration of existing data processes into a scalable Databricks target architecture

Requirements

Project Environment

The focus is not on classic BI reporting, but on building a robust data foundation for operational and analytical use cases in the energy sector.

Potential data and use case areas include, for example, asset performance, forecasting, grid data, and operational energy data. These application fields also align with the internal focus of your Energy Data & AI market. Databricks itself positions its Energy Platform for asset monitoring, forecasting, and the connection of operational, plant, and grid data, among other things.

Freelancers who have accompanied a Databricks platform from architecture and data integration to productive implementation are therefore particularly interesting.

• Several years of project experience as a Databricks Consultant, Databricks Engineer, Senior Data Engineer, or Data Platform Engineer
• Demonstrable experience with productive Databricks projects
• Very good knowledge of Apache Spark / PySpark
• Solid experience with SQL and Python
• Experience with Delta Lake and modern Lakehouse architectures
• Experience with Unity Catalog
• Confident handling of ETL/ELT processes and complex data pipelines
• Experience with a cloud platform, ideally Microsoft Azure
• Knowledge of CI/CD and automated deployment of data workloads is an advantage
• Ideally, experience in the Energy, Utilities, Grid, or Energy Trading environment

Project details

• Start: ab sofort
• Duration: 6 Monate
• Remote: remote
• Location: köln, Deutschland, DE
• Languages: German