Databricks Technical Lead

Project overview

Databricks Technical Lead l Contract | 12 Months | Outside IR35 l Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)

The role

Method Resourcing have been asked by one of our consulting partners to find a hands-on Databricks Technical Lead to design and deliver a new Azure Databricks platform for a major life sciences organisation replacing its enterprise data and analytics stack.

The programme uses AI and LLM tooling to reverse-engineer existing pipelines, business logic and reports into documented, tested code. You'll mentor three senior data engineers, own the technical architecture and stay hands-on in development.

Responsibilities

• Architect and build the platform on Azure Databricks (Delta Lake, Unity Catalog, medallion architecture)
• Use AI agents and LLMs to reverse-engineer Legacy reports, queries and transformations, and build repeatable AI-assisted processes for code generation, testing and migration
• Build automated testing and reconciliation to validate migrated data against existing reporting
• Integrate SAP, Azure Data Lake, Salesforce and other systems, and design gold-layer models for Power BI
• Set engineering standards (Workflows, CI/CD, IaC, governance) and mentor the team through to UAT and cutover

Requirements

Desirable

Contract

If this is you, send your CV to (see below) and if suitable, i'll be in touch. As you can appreciate, this role will receive a high volume of candidates so I will do my best to respond if possible.

Databricks Technical Lead l Contract | 12 Months | Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)

RSG Plc is acting as an Employment Business in relation to this vacancy.

• Extensive hands-on experience delivering and technically leading enterprise solutions on Azure Databricks
• Demonstrable use of AI, LLMs or agents to reverse-engineer data pipelines or reporting logic
• Strong Python, PySpark and SQL
• Deep knowledge of Delta Lake, Unity Catalog, Workflows and medallion architecture
• Automated validation, reconciliation and testing experience
• Azure data engineering (ADLS Gen2, ADF)
• Databricks CI/CD (Terraform, Asset Bundles or Azure DevOps)
• SAP ECC or S/4HANA integration experience
• Dimensional modelling and Power BI semantic model familiarity
• Proven team leadership while staying hands-on
• Databricks certification (Data Engineer Professional or Solutions Architect)
• Platform migration, carve-out or separation experience
• Reusable AI agent frameworks for data engineering
• Strong governance and data quality experience
• Consulting background
• outside IR35
• £900-£1,000 per day
• Remote with some travel required
• 12 month contract

Project details

• Start: ASAP
• Duration: 12 months
• Remote: onsite
• Location: Birmingham, United Kingdom, GB
• Rate: unknown-125 EUR/h
• Languages: English