Zum Inhalt springen

Diese Ausschreibung ist auf Englisch.

JobbörseOffen

Senior Databricks Engineer

Vertraulich · London, Großbritannien · Hybrid · SENIOR

  • London, Großbritannien · Hybrid
  • English

Indexiert (vor 2 Tagen) · laufend aktualisiert

Budget
63 £ / h
Start
flexible
Arbeitsmodell
Hybrid
Vertrag
Freelance / B2B
Erfahrungslevel
SENIOR
Referenz
6092291

Geforderte Skills

Nice to have:TerraformAWS IAMAWS KMSCI/CDLakehouse medallion architectureDistributed data processing

Sprachen: English

Jobbeschreibung

Project overview

Job Title: Senior Databricks Engineer-Financial Services Client

Project details

London / Manchester (3 Days/Week Onsite)

4Months+

£500 GBP/day – Inside IR35

Role Overview

We are looking for an experienced Senior Databricks Engineer to join a large-scale enterprise data engineering programme for a leading Financial Services client.

The ideal candidate will have strong hands-on experience with Databricks, PySpark, SQL, Delta Lake and AWS cloud services, along with experience building and maintaining enterprise-scale data pipelines.

The role will involve developing, configuring and maintaining Databricks Lakehouse platforms, building batch and incremental data pipelines, implementing data transformations and supporting platform deployment across multiple environments.

The successful candidate will work with Data Engineering, Cloud, DevOps, Security and Governance teams to deliver reliable, scalable and high-performance data solutions.

Experience migrating workloads from AWS-native tools to Databricks and working within financial services environments will be highly beneficial.

​

Key Responsibilities

Required Technical Skills

Good to Have

Ideal Candidate Profile

If this position is of interest to you, apply now!

  • Databricks Development & Engineering
  • Develop and maintain Databricks solutions across Development, Test, UAT and Production environments.
  • Work extensively with Databricks Notebooks, PySpark DataFrame API and SQL transformations.
  • Develop and maintain data processing pipelines using Delta Lake.
  • Build and manage Databricks Jobs, Job Flows and Workflows.
  • Work with Declarative Pipelines, including pipeline definitions and DPX files.
  • Implement Databricks Auto Loader and Structured Streaming.
  • Build both batch and incremental data pipelines.
  • Troubleshoot and resolve issues across Databricks workloads and jobs.
  • Data Engineering & Lakehouse Development
  • Develop scalable data engineering solutions using Databricks Lakehouse architecture.
  • Implement data transformations and processing using Bronze, Silver and Gold layers.
  • Develop and maintain data models for enterprise data platforms.
  • Work with distributed data processing and large-scale datasets.
  • Optimise PySpark and SQL workloads for performance.
  • Implement efficient data ingestion and processing frameworks.
  • Ensure data quality, reliability and consistency across pipelines.
  • AWS Cloud Integration
  • Integrate Databricks with AWS cloud services.
  • Work with AWS S3 for data storage and access.
  • Use AWS IAM and KMS for identity, access management and encryption.
  • Work with AWS Glue and Lambda services.
  • Support migration of workloads from AWS-native tools to Databricks.
  • Troubleshoot cloud integration and data connectivity issues.
  • Platform Configuration & Automation
  • Configure and manage Databricks workspaces, compute resources and cluster settings.
  • Work with cluster policies, job clusters and compute configurations.
  • Support infrastructure provisioning using Terraform and Infrastructure as Code (IaC).
  • Develop and maintain CI/CD pipelines for Databricks notebooks, jobs and deployments.
  • Work with Git for source control and code management.
  • Automate repetitive deployment and platform administration activities.
  • Security, Monitoring & Performance
  • Work with Unity Catalog for data governance, catalogs, schemas and permissions.
  • Implement access controls, RBAC and service principal configurations.
  • Support secure platform configurations and secrets management.
  • Monitor Databricks workloads, jobs and compute resources.
  • Implement logging, monitoring and alerting.
  • Troubleshoot production issues and improve platform reliability.
  • Optimise compute configurations and workloads for cost and performance efficiency.
  • Collaboration & Technical Delivery
  • Work closely with Data Engineering, Cloud, DevOps, Security and Application teams.
  • Support multi-environment and multi-region deployments.
  • Follow engineering standards and best practices.
  • Maintain technical documentation and deployment procedures.
  • Support platform security hardening and governance requirements.
  • Strong hands-on experience with Databricks.
  • Databricks Notebooks.
  • PySpark DataFrame API.
  • SQL transformations.
  • Delta Lake.
  • Databricks Jobs, Job Flows and Workflows.
  • Declarative Pipelines (DPX files and pipeline definitions).
  • Unity Catalog.
  • Databricks Auto Loader and Structured Streaming.
  • Experience building batch and incremental data pipelines.
  • AWS S3, IAM and KMS.
  • AWS Glue and Lambda.
  • Experience migrating AWS-native workloads to Databricks.
  • Distributed data processing and performance tuning.
  • Lakehouse medallion architecture (Bronze, Silver and Gold).
  • Git and CI/CD.
  • Terraform and Infrastructure as Code (IaC).
  • Experience with enterprise Databricks platform configuration.
  • AWS cloud platform experience.
  • Experience with multi-environment and multi-region deployments.
  • Financial services or banking domain experience.
  • Knowledge of platform security, monitoring and governance.
  • Experience supporting production Databricks environments.
  • Senior Databricks Engineer with approximately 7–10 years of relevant experience.
  • Strong hands-on experience in Databricks and enterprise data engineering.
  • Good understanding of PySpark, SQL, Delta Lake and Lakehouse architecture.
  • Experience developing batch and incremental data pipelines.
  • Strong AWS cloud experience, particularly S3, IAM and KMS.
  • Experience with Unity Catalog, Auto Loader and Structured Streaming.
  • Understanding of CI/CD, Git and data platform deployment.
  • Experience working with enterprise-scale data platforms.
  • Financial services or banking experience would be advantageous.
  • Must be comfortable working 3 days per week onsite in London or Manchester.
  • Start: flexible
  • Remote: hybrid
  • Location: London, GB
  • Rate: 63-63 EUR/h
  • Languages: English

Indexiert von der 9am-Meta-Suche. Du bist Job-Anbieter? Listing beanspruchen oder verwalten

Gefunden auf Vallum Associates

  1. JobbörseNeu

    FTTH Network Engineer

    Vertraulich· Walloon Region, Belgium· Vor Ort· SENIOR

    • Cocon
    • FTTH Network Design
    • FTTH Network Structures
    • +7

  2. JobbörseNeu

    BI Specialist

    Vertraulich· Mannheim, Germany· Hybrid· Senior

    • Power BI
    • SQL
    • Databricks
    • +5

Alle Jobs durchsuchen
Kostenlos anmelden und bewerben