Job description
Project overview
Our key client is looking for an Azure Databricks Developer with SC and NPPv3 on a 6 months+ contract.
The Azure Databricks Developer is responsible for designing, developing, testing and supporting scalable data engineering solutions using Azure Databricks and the wider Azure data platform. The role focuses on building robust data ingestion, transformation and processing pipelines that enable advanced analytics, AI, machine learning and business intelligence solutions.
Working closely with Data Architects, Data Engineers, Data Analysts, and Solution Architects, the Azure Databricks Developer helps deliver trusted, high-quality data products while ensuring adherence to security, governance and operational standards.
Requirement: NPPV3/SC clearance
Key Responsibilities
Data Engineering & Development
Design and develop data pipelines using Azure Databricks.
Build scalable ETL/ELT solutions using PySpark, Spark SQL and Delta Lake.
Develop ingestion frameworks for batch and streaming data.
Develop outbound pipelines that interface with APIs
Optimise large-scale data processing workloads for performance and cost efficiency.
Data Integration
Integrate data from:
Azure SQL Database
Azure Data Lake Storage Gen2
Microsoft Fabric
APIs and third-party SaaS platforms
On-premise systems
Event streaming platforms
Implement automated data validation and reconciliation processes.
Data Modelling
Implement bronze, silver and gold data architectures.
Optimise storage structures using Delta Lake best practices.
Platform Engineering
Configure and maintain Databricks workspaces, clusters and compute resources.
Implement CI/CD pipelines for Databricks code deployments.
Support environment promotion across development, test and production environments.
Monitor and troubleshoot platform performance issues.
Data Quality & Governance
Implement data quality controls.
Support metadata management and lineage requirements.
Ensure compliance with security, privacy and governance standards.
Contribute to data catalogue and documentation activities.
Collaboration
Work with business stakeholders to understand data requirements.
Support Data Architects in implementing target-state architectures.
Technical Skills
Essential
Azure Databricks
Apache Spark
PySpark
Spark SQL
Delta Lake
Azure Data Factory or Microsoft Fabric Data Factory
Git version control
SQL development
Python programming
Desirable
Azure Synapse Analytics
Azure Event Hubs
Kafka
Power BI
Databricks Workflows
Azure DevOps
Terraform
Unity Catalog
MLflow
Azure Machine Learning
Experience Profile
Candidates should demonstrate experience in:
Developing enterprise-scale data pipelines.
Working with large data volumes and distributed processing technologies.
Building ETL/ELT solutions within Azure environments.
Implementing data quality and monitoring controls.
Working within Agile delivery teams.
Typical Backgrounds
Data Engineer
ETL Developer
Azure Data Engineer
BI Developer with strong engineering experience
Spark Developer
Databricks Engineer
Key Competencies
Technical
Analytical problem solving
Performance optimisation
Data architecture awareness
Software engineering practices
Testing and quality assurance
#4839281 - Nasar Ali
Project details
- Start: flexible
- Location: London, GB
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on Hays UK