Required skills
- Databricks
- PySpark
- SQL
- Delta Lake
- Unity Catalog
- Databricks Auto Loader
- Structured Streaming
- AWS
- Amazon S3
- AWS IAM
- AWS Glue
- AWS Lambda
Languages: English
Job description
Project overview
You will design, build, secure, automate, and operate enterprise Databricks Lakehouse platforms for banking and financial services clients, ensuring scalability, reliability, governance, and performance across multiple environments and regions.
Key Responsibilities
- Build and manage enterprise Databricks platforms across Dev, Test, UAT, and Production environments.
- Provision and configure Databricks workspaces, clusters, compute resources, and platform settings.
- Automate infrastructure provisioning and platform configuration using Terraform and IaC.
- Develop and maintain CI/CD pipelines for Databricks deployments, notebooks, jobs, and configurations.
- Administer Unity Catalog, including catalogs, schemas, permissions, and data access controls.
- Implement security controls, RBAC, service principals, secrets management, and least-privilege access.
- Integrate Databricks with AWS/Azure services, including storage, networking, identity, and security.
- Monitor platform health, troubleshoot production issues, and implement logging, alerting, and performance optimisation.
- Optimise compute resources and automate platform operations for cost efficiency and reliability.
- Collaborate with Data Engineering, Cloud, DevOps, Security, and Governance teams to deliver secure, scalable platforms.
Requirements
Databricks & Lakehouse
AWS & Cloud
DevOps & Data Engineering
If this position is of interest to you, apply now!
- Strong hands-on experience with Databricks, including Notebooks, PySpark, SQL, Delta Lake, Jobs, Workflows, and Declarative Pipelines.
- Experience with Unity Catalog, Databricks Auto Loader, and Structured Streaming.
- Experience developing batch and incremental data pipelines.
- Strong understanding of Lakehouse architecture, including Bronze, Silver, and Gold layers.
- Good knowledge of AWS services, particularly S3, IAM, KMS, Glue, and Lambda.
- Experience migrating data workloads from AWS-native services to Databricks.
- Understanding of cloud infrastructure, security, networking, and identity management.
- Experience with Git, CI/CD pipelines, and deployment automation.
- Knowledge of Terraform or Infrastructure as Code (IaC) is desirable.
- Strong understanding of distributed data processing, data modelling, and performance optimisation.
Project details
- Start: flexible
- Location: Manchester, GB
- Languages: English
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on Vallum Associates