Data & Software Engineer -Python, SQL, Apache Kafka, Apache NiFi, Apache Spark, PySpark, Hadoop, Hiv

Project overview

Data & Software Engineer - Python, SQL, Apache Kafka, Apache NiFi, Apache Spark, PySpark, Hadoop, Hive, REST APIs, Git, GitHub, CI/CD, Microsoft Azure, Microsoft Graph, Entra ID, Grafana, Splunk, GitHub Copilot.

Seeking an experienced Data & Software Engineer to design, develop and maintain scalable data platforms, pipelines and automated compliance solutions.

Key Responsibilities

Essential Skills

Desirable

Chester/Hybrid/Contract

By applying to this job you are sending us your CV, which may contain personal information. Please refer to our Privacy Notice to understand how we process this information. In short, in order to supply you with work finding services, we will hold and process your personal data, and only with your express permission we will share this personal data with a client (or a third party working on behalf of the client) by email or by upload to the Client/third parties vendor management system.

By giving us permission to send your CV to a client, this constitutes permission to share the personal data that would be necessary to consider your application, interview you (Phone/video/face to face) and if successful hire you. Scope AT acts as an employment agency for Permanent Recruitment and an employment business for the supply of temporary workers. By applying for this job you accept the Terms and Conditions, Data Protection Policy, Privacy Notice and Disclaimers which can be found at our website.

• Build and optimise data ingestion, streaming and processing pipelines.
• Develop API integrations and data transformation solutions.
• Implement automated reconciliation, data quality and compliance controls.
• Develop and maintain production-grade Python, PySpark and SQL applications.
• Ensure data integrity, monitoring, security and auditability.
• Manage testing, CI/CD, production deployments and technical support.
• Strong Python and SQL development.
• Hands-on experience with Apache Kafka, NiFi and Spark/PySpark.
• Knowledge of Hadoop, Hive and data lake architectures.
• REST API integration and data modelling.
• Git, CI/CD, automated testing and production support.
• Experience building scalable, resilient data engineering solutions.
• Financial services or regulated industry experience.
• Azure, Microsoft Graph and Entra ID.
• Voice recording or collaboration platforms.
• Grafana, Splunk and AI-assisted development tools.

Project details

• Start: ASAP
• Location: Chester, UK, GB