Zum Inhalt springen

Diese Ausschreibung ist auf Englisch.

JobbörseOffen

Data Engineer

Vertraulich · Manchester, Großbritannien · Senior

  • Manchester, Großbritannien
  • English

Indexiert (gestern) · laufend aktualisiert

Start
flexible
Vertrag
Freelance / B2B
Erfahrungslevel
Senior
Referenz
6093139

Geforderte Skills

Nice to have:TerraformGitCI/CDAWS KMS

Sprachen: English

Jobbeschreibung

Project overview

You will design, build, secure, automate, and operate enterprise Databricks Lakehouse platforms for banking and financial services clients, ensuring scalability, reliability, governance, and performance across multiple environments and regions.

Key Responsibilities

  • Build and manage enterprise Databricks platforms across Dev, Test, UAT, and Production environments.
  • Provision and configure Databricks workspaces, clusters, compute resources, and platform settings.
  • Automate infrastructure provisioning and platform configuration using Terraform and IaC.
  • Develop and maintain CI/CD pipelines for Databricks deployments, notebooks, jobs, and configurations.
  • Administer Unity Catalog, including catalogs, schemas, permissions, and data access controls.
  • Implement security controls, RBAC, service principals, secrets management, and least-privilege access.
  • Integrate Databricks with AWS/Azure services, including storage, networking, identity, and security.
  • Monitor platform health, troubleshoot production issues, and implement logging, alerting, and performance optimisation.
  • Optimise compute resources and automate platform operations for cost efficiency and reliability.
  • Collaborate with Data Engineering, Cloud, DevOps, Security, and Governance teams to deliver secure, scalable platforms.

Requirements

Databricks & Lakehouse

AWS & Cloud

DevOps & Data Engineering

If this position is of interest to you, apply now!

  • Strong hands-on experience with Databricks, including Notebooks, PySpark, SQL, Delta Lake, Jobs, Workflows, and Declarative Pipelines.
  • Experience with Unity Catalog, Databricks Auto Loader, and Structured Streaming.
  • Experience developing batch and incremental data pipelines.
  • Strong understanding of Lakehouse architecture, including Bronze, Silver, and Gold layers.
  • Good knowledge of AWS services, particularly S3, IAM, KMS, Glue, and Lambda.
  • Experience migrating data workloads from AWS-native services to Databricks.
  • Understanding of cloud infrastructure, security, networking, and identity management.
  • Experience with Git, CI/CD pipelines, and deployment automation.
  • Knowledge of Terraform or Infrastructure as Code (IaC) is desirable.
  • Strong understanding of distributed data processing, data modelling, and performance optimisation.

Project details

  • Start: flexible
  • Location: Manchester, GB
  • Languages: English

Indexiert von der 9am-Meta-Suche. Du bist Job-Anbieter? Listing beanspruchen oder verwalten

Gefunden auf Vallum Associates

Alle Jobs durchsuchen
Kostenlos anmelden und bewerben