Data Engineer (Azure Databricks and Scala)

Project overview

Domain Requirement: Financial Services (Mandatory)

Role Overview

We are seeking a highly skilled Senior Data Engineer with a deep background in the Financial Services domain to design, build, and optimize complex data engineering solutions. In this role, you will leverage Azure Databricks, Azure Data Factory, and Scala/PySpark to build robust, scalable data pipelines handling multi-million-record datasets. You will work closely with cross-functional teams, participate in customer interviews, and ensure our financial data platform meets rigorous standards for performance, scalability, and security.

Key Responsibilities

Requirements & Key Skills

Mandatory Requirements

Nice-to-Have Skills

• Data Pipeline Engineering: Design, build, and maintain enterprise-grade batch and streaming data software solutions using Azure Databricks, Azure Data Factory (ADF), and Scala.
• Performance Optimization: Analyze, tune, and optimize data pipeline processing performance across complex, large-scale financial datasets (multi-million records).
• Cross-Functional Collaboration: Collaborate closely with product managers, data architects, and business stakeholders to translate complex financial business requirements into scalable technical solutions.
• Customer & Stakeholder Interaction: Participate in customer and stakeholder discovery interviews to clarify technical requirements, propose architectures, and gather feedback.
• Project Management & Communication: Proactively communicate project milestones, technical risks, and project status; escalate critical blockers to management in a timely manner.
• Quality & Governance: Adhere to best practices in data governance, security, and compliance specific to the financial services industry.
• Domain Experience: Proven track record of working within the Financial Services/Banking/Fintech domain (eg, risk, compliance, transactional systems, reporting).
• Core Technical Experience: 6 to 8+ years of hands-on data engineering experience, with expertise in designing and implementing end-to-end data pipelines using Azure Databricks and Azure Data Factory (ADF).
• Large-Scale Data Optimization: Demonstrated expertise in performance tuning, resource allocation, and optimizing Spark queries on large-scale datasets (multi-million/billion records).
• Language Proficiency: Strong programming background; proficiency or development experience in Scala (or heavy PySpark experience with a willingness to write Scala).
• Cloud Platform Knowledge: Solid operational understanding of the broader Azure Cloud ecosystem (eg, Azure Data Lake Storage, Azure Key Vault, Azure Synapse/SQL).
• Soft Skills: Exceptional problem-solving abilities, strong verbal/written communication, and a collaborative team mindset.
• Certifications: Databricks Certified Data Engineer Professional or Azure Data Engineer Associate (DP-203).
• Methodology: Experience working in an Agile/Scrum delivery environment with an understanding of Agile principles and practices.
• DevOps: Familiarity with CI/CD pipelines (eg, Azure DevOps, GitHub Actions) for automated testing, deployment, and data pipeline orchestration.

Project details

• Start: flexible
• Location: London, UK, GB