Required skills
- Databricks
- PySpark
- SQL
- Delta Lake
- Databricks Auto Loader
- Spark Structured Streaming
- Databricks Workflows
- Unity Catalog
- AWS S3
- AWS Glue
- AWS Lambda
- Git
Languages: English
Job description
Project overview
Job Title: Senior Databricks Engineer-Financial Services Client
Project details
London / Manchester (3 Days/Week Onsite)
4Months+
£500 GBP/day – Inside IR35
Role Overview
We are looking for an experienced Senior Databricks Engineer to join a large-scale enterprise data engineering programme for a leading Financial Services client.
The ideal candidate will have strong hands-on experience with Databricks, PySpark, SQL, Delta Lake and AWS cloud services, along with experience building and maintaining enterprise-scale data pipelines.
The role will involve developing, configuring and maintaining Databricks Lakehouse platforms, building batch and incremental data pipelines, implementing data transformations and supporting platform deployment across multiple environments.
The successful candidate will work with Data Engineering, Cloud, DevOps, Security and Governance teams to deliver reliable, scalable and high-performance data solutions.
Experience migrating workloads from AWS-native tools to Databricks and working within financial services environments will be highly beneficial.
Key Responsibilities
Required Technical Skills
Good to Have
Ideal Candidate Profile
If this position is of interest to you, apply now!
- Databricks Development & Engineering
- Develop and maintain Databricks solutions across Development, Test, UAT and Production environments.
- Work extensively with Databricks Notebooks, PySpark DataFrame API and SQL transformations.
- Develop and maintain data processing pipelines using Delta Lake.
- Build and manage Databricks Jobs, Job Flows and Workflows.
- Work with Declarative Pipelines, including pipeline definitions and DPX files.
- Implement Databricks Auto Loader and Structured Streaming.
- Build both batch and incremental data pipelines.
- Troubleshoot and resolve issues across Databricks workloads and jobs.
- Data Engineering & Lakehouse Development
- Develop scalable data engineering solutions using Databricks Lakehouse architecture.
- Implement data transformations and processing using Bronze, Silver and Gold layers.
- Develop and maintain data models for enterprise data platforms.
- Work with distributed data processing and large-scale datasets.
- Optimise PySpark and SQL workloads for performance.
- Implement efficient data ingestion and processing frameworks.
- Ensure data quality, reliability and consistency across pipelines.
- AWS Cloud Integration
- Integrate Databricks with AWS cloud services.
- Work with AWS S3 for data storage and access.
- Use AWS IAM and KMS for identity, access management and encryption.
- Work with AWS Glue and Lambda services.
- Support migration of workloads from AWS-native tools to Databricks.
- Troubleshoot cloud integration and data connectivity issues.
- Platform Configuration & Automation
- Configure and manage Databricks workspaces, compute resources and cluster settings.
- Work with cluster policies, job clusters and compute configurations.
- Support infrastructure provisioning using Terraform and Infrastructure as Code (IaC).
- Develop and maintain CI/CD pipelines for Databricks notebooks, jobs and deployments.
- Work with Git for source control and code management.
- Automate repetitive deployment and platform administration activities.
- Security, Monitoring & Performance
- Work with Unity Catalog for data governance, catalogs, schemas and permissions.
- Implement access controls, RBAC and service principal configurations.
- Support secure platform configurations and secrets management.
- Monitor Databricks workloads, jobs and compute resources.
- Implement logging, monitoring and alerting.
- Troubleshoot production issues and improve platform reliability.
- Optimise compute configurations and workloads for cost and performance efficiency.
- Collaboration & Technical Delivery
- Work closely with Data Engineering, Cloud, DevOps, Security and Application teams.
- Support multi-environment and multi-region deployments.
- Follow engineering standards and best practices.
- Maintain technical documentation and deployment procedures.
- Support platform security hardening and governance requirements.
- Strong hands-on experience with Databricks.
- Databricks Notebooks.
- PySpark DataFrame API.
- SQL transformations.
- Delta Lake.
- Databricks Jobs, Job Flows and Workflows.
- Declarative Pipelines (DPX files and pipeline definitions).
- Unity Catalog.
- Databricks Auto Loader and Structured Streaming.
- Experience building batch and incremental data pipelines.
- AWS S3, IAM and KMS.
- AWS Glue and Lambda.
- Experience migrating AWS-native workloads to Databricks.
- Distributed data processing and performance tuning.
- Lakehouse medallion architecture (Bronze, Silver and Gold).
- Git and CI/CD.
- Terraform and Infrastructure as Code (IaC).
- Experience with enterprise Databricks platform configuration.
- AWS cloud platform experience.
- Experience with multi-environment and multi-region deployments.
- Financial services or banking domain experience.
- Knowledge of platform security, monitoring and governance.
- Experience supporting production Databricks environments.
- Senior Databricks Engineer with approximately 7–10 years of relevant experience.
- Strong hands-on experience in Databricks and enterprise data engineering.
- Good understanding of PySpark, SQL, Delta Lake and Lakehouse architecture.
- Experience developing batch and incremental data pipelines.
- Strong AWS cloud experience, particularly S3, IAM and KMS.
- Experience with Unity Catalog, Auto Loader and Structured Streaming.
- Understanding of CI/CD, Git and data platform deployment.
- Experience working with enterprise-scale data platforms.
- Financial services or banking experience would be advantageous.
- Must be comfortable working 3 days per week onsite in London or Manchester.
- Start: flexible
- Remote: hybrid
- Location: London, GB
- Rate: 63-63 EUR/h
- Languages: English
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on Vallum Associates