Skip to content
Job boardOpen

Databricks Architect

Confidential · London, United Kingdom · Hybrid · Senior

  • London, United Kingdom · Hybrid
  • English

Indexed (2 days ago) · refreshed constantly

Budget
£78 / h
Start
flexible
Work mode
Hybrid
Contract
Freelance / B2B
Seniority
Senior
Reference
6092269

Required skills

Nice to have:Microsoft AzureCI/CDGit

Languages: English

Job description

Project overview

Job Title: Databricks Architect-Financial Services Client

Project details

London / Manchester (3 Days/Week Onsite)

4 Months+

£620 GBP/day – Inside IR35

Role Overview

We are looking for an experienced Databricks Architect to join a large-scale enterprise data platform programme for a leading Financial Services client.

The ideal candidate will have strong expertise in Databricks, Lakehouse architecture, cloud data platforms and enterprise data engineering, along with experience designing and implementing scalable, secure and high-performance data solutions.

The role will involve providing architectural direction for enterprise Databricks Lakehouse platforms, defining technical standards, designing platform configurations and ensuring effective integration with cloud services such as AWS and Azure.

The successful candidate will work closely with Data Engineering, Cloud, DevOps, Security and Governance teams to deliver reliable, secure and scalable data platform solutions across multiple environments.

Experience working within banking or financial services environments will be highly beneficial.

​

Key Responsibilities

Required Technical Skills

Good to Have

Ideal Candidate Profile

If this position is of interest to you, apply now!

  • Databricks Architecture & Platform Design
  • Define and design enterprise Databricks Lakehouse architecture across Development, Test, UAT and Production environments.
  • Provide architectural direction for Databricks workspace provisioning, compute resources and platform configurations.
  • Define platform architecture standards, design principles and implementation best practices.
  • Design scalable and secure Databricks solutions supporting enterprise data engineering workloads.
  • Guide the selection and configuration of compute resources, cluster policies and job clusters.
  • Ensure platform architecture supports performance, scalability, availability and cost efficiency.
  • Lakehouse Architecture & Data Engineering
  • Design and oversee Lakehouse architecture using Delta Lake.
  • Define data architecture and modelling standards using Bronze, Silver and Gold layers.
  • Provide guidance on batch and incremental data processing pipelines.
  • Oversee the use of Databricks Notebooks, PySpark and SQL transformations.
  • Design and guide the implementation of Databricks Jobs, Workflows and Declarative Pipelines.
  • Define architecture patterns for Databricks Auto Loader and Structured Streaming.
  • Ensure efficient distributed data processing and workload optimisation.
  • Cloud Integration & Infrastructure
  • Design integration between Databricks and cloud platforms such as AWS and Azure.
  • Define architecture for cloud storage, networking, identity and security integrations.
  • Provide guidance on AWS services including S3, IAM, KMS, Glue and Lambda.
  • Design infrastructure provisioning and platform configuration using Terraform and Infrastructure as Code (IaC).
  • Support migration of workloads from AWS-native services to Databricks.
  • Define standards for multi-environment and multi-region platform deployments.
  • Security, Governance & Platform Management
  • Define security architecture and governance standards for Databricks platforms.
  • Design and oversee Unity Catalog implementation, including catalogs, schemas, permissions and access controls.
  • Establish Role-Based Access Control (RBAC), service principals and secrets management.
  • Ensure least-privilege access and secure platform configurations.
  • Define platform monitoring, logging, alerting and operational standards.
  • Ensure compliance with enterprise security and governance requirements.
  • Technical Leadership & Delivery
  • Provide technical leadership and architectural guidance to engineering teams.
  • Work closely with Data Engineering, Cloud, DevOps, Security and Application teams.
  • Review technical designs and ensure alignment with enterprise architecture standards.
  • Guide CI/CD implementation for Databricks configurations, notebooks, jobs and deployments.
  • Drive performance optimisation and platform cost efficiency.
  • Maintain architectural documentation, technical standards and implementation guidelines.
  • Strong experience in Databricks architecture and enterprise Lakehouse platforms.
  • Databricks Notebooks, PySpark and SQL.
  • Delta Lake and Lakehouse architecture.
  • Unity Catalog and data governance.
  • Databricks Jobs, Job Flows and Workflows.
  • Declarative Pipelines and pipeline definitions (DPX files).
  • Databricks Auto Loader and Structured Streaming.
  • AWS services – S3, IAM, KMS, Glue and Lambda.
  • Terraform and Infrastructure as Code (IaC).
  • CI/CD, Git and deployment automation.
  • Distributed data processing and performance optimisation.
  • Multi-environment and multi-region platform architecture.
  • Financial services or banking domain experience.
  • Experience migrating AWS-native workloads to Databricks.
  • Experience with enterprise cloud architecture.
  • Knowledge of Azure cloud services.
  • Experience with platform security hardening and enterprise governance.
  • Experience designing large-scale data platforms.
  • Experience with automated monitoring and operational frameworks.
  • Experienced Databricks Architect with strong enterprise data platform experience.
  • Strong background in designing and implementing Databricks Lakehouse solutions.
  • Hands-on understanding of PySpark, Delta Lake, SQL and Databricks workflows.
  • Experience defining cloud infrastructure, security and governance architecture.
  • Strong knowledge of AWS and Terraform/IaC.
  • Experience providing technical leadership and architectural direction.
  • Ability to work with cross-functional engineering and cloud teams.
  • Financial services or banking experience would be advantageous.
  • Must be comfortable working 3 days per week onsite in London or Manchester.
  • Start: flexible
  • Remote: hybrid
  • Location: London, GB
  • Rate: 78-78 EUR/h
  • Languages: English

Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing

Found on Vallum Associates

  1. Job boardNew

    Data Analyst

    Unitech Solutions· Lyon, France· Hybrid· SENIOR· €50 / h

    • Power BI
    • DAX
    • Power Query
    • +4

Browse all jobs
Sign up free to apply