Data Pipeline Architect | Outside IR35

Project overview

Data Pipeline Architect - Outside IR35

SR2 is looking for a Data Pipeline Architect to support an open-source data integration programme. Combining architectural ownership with hands-on development, you'll build reusable pipelines that connect systems and transform complex datasets reliably.

The Role

You'll shape the architecture and delivery of a portable data transformation platform, using Apache NiFi alongside Python and Java. The platform must operate across different hosting environments, with a particular focus on RDF data processing and SPARQL.

This role suits someone who can turn emerging requirements into practical designs and take those designs through development, testing, deployment and operational support.

Key Responsibilities

Essential Skills and Experience

Desirable Experience

The Details

If you have strong Apache NiFi experience alongside RDF, SPARQL, Python and Java, get in touch with SR2 or apply with your latest CV.

• Define the platform architecture, establishing reusable patterns for ingestion, transformation, validation and delivery across batch and streaming workloads.
• Develop and maintain Apache NiFi flows connecting APIs, databases, files, object storage, messaging platforms and SPARQL endpoints.
• Build RDF processing pipelines, map source data into agreed semantic models, and develop SPARQL queries and updates that preserve data meaning and quality.
• Write Python and Java components, including custom integrations and NiFi extensions where required.
• Embed automated testing, monitoring, version control and repeatable deployment, with robust handling of retries, duplicates, failures and recovery.
• Work with engineers and stakeholders to refine requirements, document technical decisions and deliver capabilities against agreed milestones.
• Proven experience designing and delivering data integration or ETL platforms using Apache NiFi.
• Practical project experience with RDF triples and SPARQL, including semantic modelling, vocabularies, identifiers and named graphs.
• Hands-on development capability in both Python and Java.
• Experience building reusable pipelines across multiple source systems and data formats.
• Strong understanding of reliable processing, including incremental loads, idempotency, duplicate handling and failure recovery.
• Experience implementing data validation, automated tests, operational monitoring and CI/CD.
• Understanding of secure integration, including authentication, authorisation, TLS and credential management.
• Ability to assess technology trade-offs, turn incomplete requirements into clear specifications and explain designs to technical and non-technical stakeholders.
• Apache Jena and Apache Jena Fuseki.
• Apache Kafka and integration with NiFi.
• SHACL or comparable RDF validation approaches.
• Custom NiFi processor or controller service development.
• SPARQL optimisation and performance tuning for large RDF datasets.
• Docker, Kubernetes and open-source development or contributions.
• Contract role
• Duration: Initially until the end of December 2026
• Rate: Up to £475p/d
• IR35: Outside IR35
• Location: Remote

Project details

• Start: flexible
• Duration: 3 months
• Remote: remote
• Location: London, United Kingdom, GB