Databricks Data Engineer (AWS, Delta Lake, Streaming)

Project overview

Contract Databricks Data Engineer (2–5 yrs) needed in Gurugram/Noida to build Databricks pipelines using Python notebooks and Workflows, implement near real-time S3 monitor-based streaming ingestion on AWS, and model data using Delta Lake medallion (Bronze/Silver/Gold) with governance via Unity Catalog. The role will serve curated datasets through Databricks SQL Warehouse to downstream services (BFFs) while enforcing data quality, study-level isolation, and GxP-compliant audit trails. Must be strong in Databricks, Python, SQL, AWS (S3/Secrets Manager/CloudWatch), streaming ingestion, and Git/Jenkins CI/CD.

Responsibilities

• Build Databricks pipelines (Python notebooks + Workflows) ingesting from Rave, IWRS, CSS, SAE, protocol deviations, etc.
• Implement real-time S3 monitor-based streaming ingestion.
• Model data across Bronze ? Silver ? Gold in Delta Lake, governed by Unity Catalog.
• Serve curated data via Databricks SQL Warehouse to BFFs.
• Enforce data quality, study-level isolation, audit trails (GxP).
• 2+ yrs data engineering; strong Databricks, Python, SQL.
• AWS (S3, Secrets Manager, CloudWatch).
• Streaming ingestion; Git + Jenkins CI/CD.
• Unity Catalog, Databricks Asset Bundles, Vector Search / Model Serving.

Requirements

• Databricks (notebooks, Workflows)
• Python
• SQL
• Data engineering (2+ years)
• Delta Lake
• Medallion architecture (Bronze/Silver/Gold)
• AWS S3
• AWS Secrets Manager
• AWS CloudWatch
• Streaming ingestion (Databricks/Spark streaming patterns)
• Git
• Jenkins CI/CD
• Data quality enforcement
• Study-level data isolation
• Audit trails / GxP compliance
• AI-driven development

Project details

• Start: flexible
• Remote: hybrid
• Location: India, IN