QA Lead

Project overview

This contract QA Lead role focuses on testing backend APIs and evaluating LLM-enabled product capabilities, with work involving Product, Clinical, and R&D partners. The core requirements are 10+ years in backend/API test automation, performance testing, CI/CD, basic AWS knowledge, hands-on testing tools, LLM/ML validation, and disciplined PDLM/QMS documentation. Azure DevOps, FHIR/HL7, healthcare privacy and security, and frontend testing are bonuses rather than core requirements. The structured experience band is 10–15 years, while the description specifies 10+ years; these are broadly aligned.

The role is based in Bengaluru, is contract, and does not specify an onsite, hybrid, or remote work mode.

Responsibilities

• Design, automate, and execute API test suites (functional, contract, performance, resilience)
• Define LLM evaluation strategies (performance testing, golden sets, guardrails, hallucination/consistency checks, relevance & latency KPIs); build automated eval pipelines tied to releases.
• Create PDLM deliverables: verification plans/protocols, test reports, risk-based coverage, V&V trace matrices to requirements and use cases.
• Integrate test infrastructure with Azure DevOps (CI/CD, test orchestration, quality gates, dashboards); enforce build-time gates for API and LLM quality.
• Partner with Product, Clinical, and R&D to refine requirements and acceptance criteria; ensure summaries render correctly for different user roles and workflows.
• 10+ years in backend/API test automation (REST, JSON, auth), performance testing, and CI/CD + Mandatory basic working knowledge with AWS Cloud
• Hands-on with Python/TypeScript, Postman/Newman or REST-assured, PyTest/Jest, k6/JMeter, Docker.
• Experience validating ML/LLM systems (eval datasets, metrics, prompt/response testing, safety/guardrails).
• Strong documentation and traceability discipline for regulated product lifecycles (PDLM/QMS)
• Bonus: Azure DevOps pipelines, FHIR/HL7 familiarity, healthcare data privacy/security.
• Bonus: Frontend testing capabilities are preferred, but not part of the PDLM requirements.

Requirements

• 10+ years of backend/API test automation, including REST, JSON, and authentication
• API functional, contract, performance, and resilience testing
• Performance testing
• CI/CD test integration, orchestration, quality gates, and dashboards
• Basic working knowledge of AWS Cloud
• Hands-on Python or TypeScript
• Hands-on API testing tools: Postman/Newman or REST-assured
• Hands-on test frameworks: PyTest or Jest
• Hands-on performance tools: k6 or JMeter
• Docker
• ML/LLM system validation, including evaluation datasets, metrics, prompt/response testing, and safety/guardrails
• PDLM/QMS documentation, verification plans/protocols, test reporting, risk-based coverage, and V&V traceability

Project details

• Start: flexible
• Remote: hybrid
• Location: India, IN