Data Engineer - London - ETRM - Databricks | AZURE | Python

Project overview

We are looking for a highly skilled?Senior Data Engineer?with strong analytical skill, problem-solving capabilities and hands-on experience in building scalable data solutions to work on a project at a major Engery Trading organisation based in London.

The ideal candidate will have expertise in?Python, Databricks (PySpark), streaming architectures, data ingestion frameworks, and?Performance/Cost Optimization Technique, along with experience working with'sAP, ETRM data sources would be an added advantage.?The candidate will be responsible for designing and implementing scalable data pipelines and enabling real-time, batch analytics and across the program.

Responsibilities

• Design and implement scalable data pipelines using?Databricks and Delta Live Tables (DLT).
• Develop and manage?streaming data solutions?for Real Time analytics and decision-making.
• Ingest data from various sources-including?SAP, ETRM systems using Qlik Replicate?and?ADF?to ingest data from various sources ensuring data quality and consistency.
• Perform?data modelling?and optimize data structures for analytics and reporting.
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Requirements

• Strong experience with?Databricks- DLT, Delta Lake concepts, UC governance.
• Solid understanding of?streaming technologies?(eg, Spark Structured Streaming, Autoloader).
• Programming skills in?Python,?SQL, or?Scala.
• Proficiency in?data modelling, ETL/ELT processes, and data architecture.
• Experience in MS Copilot, AI Foundry, Databricks (Agentbricks, Genie)
• Strong analytical background with problem-solving skills.
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
• Hands-on expertise in?ADF, and Qlik Replicate?for data ingestion and replication.
• Experience working in?Azure cloud environments.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Project details

• Start: ASAP
• Duration: 6 months initial
• Location: London/Hybrid, UK, GB