Senior Data Engineer - Contract

Project overview

Senior Data Engineer - Airflow/Azure/PySpark

£750 per day | Inside IR35 | 6-month contract | London | Urgent requirement

We're looking for an experienced Senior Data Engineer to join a specialist financial services business on an urgent 6-month contract.

You'll play a key role in developing and enhancing a business-critical data platform, building scalable data pipelines and delivering reliable, high-quality data to support complex analytics and reporting requirements.

This is a hands-on engineering role suited to someone who has worked with high-volume data, complex processing requirements and time-sensitive analytics. You'll work closely with engineering teams and business stakeholders, taking ownership of solutions from design through to delivery.

Key responsibilities

Key skills and experience

Desirable experience

Contract details

This is an urgent requirement, and candidates available to start immediately or at short notice are strongly preferred.

If you have strong Airflow, Azure and PySpark or Databricks experience and are available for your next contract, please get in touch for further details.

• Design, build and maintain scalable data pipelines using Apache Airflow, Azure and PySpark or Databricks.
• Develop and optimise data processing workflows to support complex analytical and reporting requirements.
• Work with large, complex datasets, ensuring data is accurate, reliable and available for downstream consumption.
• Build and maintain data products within a modern cloud-based data platform.
• Implement automated data quality checks, monitoring and robust operational processes.
• Identify performance bottlenecks and improve the efficiency, reliability and scalability of existing pipelines.
• Work closely with technical teams and business stakeholders to translate requirements into effective data engineering solutions.
• Contribute to architectural decisions and promote best practices across data engineering, orchestration and platform development.
• Strong commercial experience as a Senior Data Engineer or in a similar role.
• Hands-on experience with Apache Airflow.
• Strong experience working with Microsoft Azure data platforms.
• Strong experience with either PySpark or Databricks.
• Proven experience designing, developing and maintaining production-grade data pipelines.
• Experience processing large datasets and building reliable data workflows for analytical use cases.
• Strong understanding of data quality, pipeline monitoring, performance optimisation and production support.
• Experience working in technically complex, data-intensive environments.
• Strong communication skills and the ability to work collaboratively with engineering teams and business stakeholders.
• Experience supporting Real Time, near-Real Time or time-sensitive analytics.
• Background in insurance, banking, financial services, sports betting, trading or other data-intensive industries.
• Experience working with cloud-based Lakehouse architectures and modern data platforms.
• Knowledge of data orchestration, distributed processing and large-scale data transformation.
• Rate: £750 per day
• IR35: Inside IR35
• Duration: Initial 6 months
• Location: London
• Start date: Immediate/urgent

Project details

• Start: Immediate
• Duration: 6 months
• Location: London, UK, GB
• Rate: 94-94 EUR/h