Skip to content
Job boardOpen

Data Engineer

Confidential · Manchester, United Kingdom · Senior

  • Manchester, United Kingdom
  • English

Indexed (yesterday) · refreshed constantly

Start
flexible
Contract
Freelance / B2B
Seniority
Senior
Reference
6093139

Required skills

Nice to have:TerraformGitCI/CDAWS KMS

Languages: English

Job description

Project overview

You will design, build, secure, automate, and operate enterprise Databricks Lakehouse platforms for banking and financial services clients, ensuring scalability, reliability, governance, and performance across multiple environments and regions.

Key Responsibilities

  • Build and manage enterprise Databricks platforms across Dev, Test, UAT, and Production environments.
  • Provision and configure Databricks workspaces, clusters, compute resources, and platform settings.
  • Automate infrastructure provisioning and platform configuration using Terraform and IaC.
  • Develop and maintain CI/CD pipelines for Databricks deployments, notebooks, jobs, and configurations.
  • Administer Unity Catalog, including catalogs, schemas, permissions, and data access controls.
  • Implement security controls, RBAC, service principals, secrets management, and least-privilege access.
  • Integrate Databricks with AWS/Azure services, including storage, networking, identity, and security.
  • Monitor platform health, troubleshoot production issues, and implement logging, alerting, and performance optimisation.
  • Optimise compute resources and automate platform operations for cost efficiency and reliability.
  • Collaborate with Data Engineering, Cloud, DevOps, Security, and Governance teams to deliver secure, scalable platforms.

Requirements

Databricks & Lakehouse

AWS & Cloud

DevOps & Data Engineering

If this position is of interest to you, apply now!

  • Strong hands-on experience with Databricks, including Notebooks, PySpark, SQL, Delta Lake, Jobs, Workflows, and Declarative Pipelines.
  • Experience with Unity Catalog, Databricks Auto Loader, and Structured Streaming.
  • Experience developing batch and incremental data pipelines.
  • Strong understanding of Lakehouse architecture, including Bronze, Silver, and Gold layers.
  • Good knowledge of AWS services, particularly S3, IAM, KMS, Glue, and Lambda.
  • Experience migrating data workloads from AWS-native services to Databricks.
  • Understanding of cloud infrastructure, security, networking, and identity management.
  • Experience with Git, CI/CD pipelines, and deployment automation.
  • Knowledge of Terraform or Infrastructure as Code (IaC) is desirable.
  • Strong understanding of distributed data processing, data modelling, and performance optimisation.

Project details

  • Start: flexible
  • Location: Manchester, GB
  • Languages: English

Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing

Found on Vallum Associates

Browse all jobs
Sign up free to apply