Required skills
- Databricks
- Lakehouse Architecture
- Delta Lake
- PySpark
- SQL
- Unity Catalog
- Databricks Workflows
- Declarative Pipelines
- Databricks Auto Loader
- Structured Streaming
- AWS
- Terraform
Languages: English
Job description
Project overview
Job Title: Databricks Architect-Financial Services Client
Project details
London / Manchester (3 Days/Week Onsite)
4 Months+
£620 GBP/day – Inside IR35
Role Overview
We are looking for an experienced Databricks Architect to join a large-scale enterprise data platform programme for a leading Financial Services client.
The ideal candidate will have strong expertise in Databricks, Lakehouse architecture, cloud data platforms and enterprise data engineering, along with experience designing and implementing scalable, secure and high-performance data solutions.
The role will involve providing architectural direction for enterprise Databricks Lakehouse platforms, defining technical standards, designing platform configurations and ensuring effective integration with cloud services such as AWS and Azure.
The successful candidate will work closely with Data Engineering, Cloud, DevOps, Security and Governance teams to deliver reliable, secure and scalable data platform solutions across multiple environments.
Experience working within banking or financial services environments will be highly beneficial.
Key Responsibilities
Required Technical Skills
Good to Have
Ideal Candidate Profile
If this position is of interest to you, apply now!
- Databricks Architecture & Platform Design
- Define and design enterprise Databricks Lakehouse architecture across Development, Test, UAT and Production environments.
- Provide architectural direction for Databricks workspace provisioning, compute resources and platform configurations.
- Define platform architecture standards, design principles and implementation best practices.
- Design scalable and secure Databricks solutions supporting enterprise data engineering workloads.
- Guide the selection and configuration of compute resources, cluster policies and job clusters.
- Ensure platform architecture supports performance, scalability, availability and cost efficiency.
- Lakehouse Architecture & Data Engineering
- Design and oversee Lakehouse architecture using Delta Lake.
- Define data architecture and modelling standards using Bronze, Silver and Gold layers.
- Provide guidance on batch and incremental data processing pipelines.
- Oversee the use of Databricks Notebooks, PySpark and SQL transformations.
- Design and guide the implementation of Databricks Jobs, Workflows and Declarative Pipelines.
- Define architecture patterns for Databricks Auto Loader and Structured Streaming.
- Ensure efficient distributed data processing and workload optimisation.
- Cloud Integration & Infrastructure
- Design integration between Databricks and cloud platforms such as AWS and Azure.
- Define architecture for cloud storage, networking, identity and security integrations.
- Provide guidance on AWS services including S3, IAM, KMS, Glue and Lambda.
- Design infrastructure provisioning and platform configuration using Terraform and Infrastructure as Code (IaC).
- Support migration of workloads from AWS-native services to Databricks.
- Define standards for multi-environment and multi-region platform deployments.
- Security, Governance & Platform Management
- Define security architecture and governance standards for Databricks platforms.
- Design and oversee Unity Catalog implementation, including catalogs, schemas, permissions and access controls.
- Establish Role-Based Access Control (RBAC), service principals and secrets management.
- Ensure least-privilege access and secure platform configurations.
- Define platform monitoring, logging, alerting and operational standards.
- Ensure compliance with enterprise security and governance requirements.
- Technical Leadership & Delivery
- Provide technical leadership and architectural guidance to engineering teams.
- Work closely with Data Engineering, Cloud, DevOps, Security and Application teams.
- Review technical designs and ensure alignment with enterprise architecture standards.
- Guide CI/CD implementation for Databricks configurations, notebooks, jobs and deployments.
- Drive performance optimisation and platform cost efficiency.
- Maintain architectural documentation, technical standards and implementation guidelines.
- Strong experience in Databricks architecture and enterprise Lakehouse platforms.
- Databricks Notebooks, PySpark and SQL.
- Delta Lake and Lakehouse architecture.
- Unity Catalog and data governance.
- Databricks Jobs, Job Flows and Workflows.
- Declarative Pipelines and pipeline definitions (DPX files).
- Databricks Auto Loader and Structured Streaming.
- AWS services – S3, IAM, KMS, Glue and Lambda.
- Terraform and Infrastructure as Code (IaC).
- CI/CD, Git and deployment automation.
- Distributed data processing and performance optimisation.
- Multi-environment and multi-region platform architecture.
- Financial services or banking domain experience.
- Experience migrating AWS-native workloads to Databricks.
- Experience with enterprise cloud architecture.
- Knowledge of Azure cloud services.
- Experience with platform security hardening and enterprise governance.
- Experience designing large-scale data platforms.
- Experience with automated monitoring and operational frameworks.
- Experienced Databricks Architect with strong enterprise data platform experience.
- Strong background in designing and implementing Databricks Lakehouse solutions.
- Hands-on understanding of PySpark, Delta Lake, SQL and Databricks workflows.
- Experience defining cloud infrastructure, security and governance architecture.
- Strong knowledge of AWS and Terraform/IaC.
- Experience providing technical leadership and architectural direction.
- Ability to work with cross-functional engineering and cloud teams.
- Financial services or banking experience would be advantageous.
- Must be comfortable working 3 days per week onsite in London or Manchester.
- Start: flexible
- Remote: hybrid
- Location: London, GB
- Rate: 78-78 EUR/h
- Languages: English
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on Vallum Associates