Skip to content
Job boardOpen

Senior Automation QA Engineer

AVENGA (Agencja Pracy, nr KRAZ: 8448) · Remote, Poland · Remote · SENIOR

  • Remote, Poland · Remote
  • English · Polish

Indexed (2 days ago) · refreshed constantly

Budget
zł 120–130 / h
Start
ASAP
Work mode
Remote
Contract
Freelance / B2B
Seniority
SENIOR
Reference
CIAJYBS2

Required skills

Nice to have:AWS LambdaAmazon DynamoDBDockerAmazon CloudWatchLangGraphBurp SuiteOWASP ZAPPromptfooDeepEvalRagas

Languages: English · Polish

Job description

We are looking for a Senior Automation QA Engineer to join a project focused on developing an enterprise AI agent development platform. The platform enables teams to build, deploy, and manage AI agents in a secure, scalable cloud environment. It provides isolated workspaces, browser-based development environments, automated agent deployment, an agent marketp

Project overview

We are looking for a Senior Automation QA Engineer to join a project focused on developing an enterprise AI agent development platform. The platform enables teams to build, deploy, and manage AI agents in a secure, scalable cloud environment. It provides isolated workspaces, browser-based development environments, automated agent deployment, an agent marketplace, and real-time monitoring.

  • Define and maintain the test strategy and test pyramid in collaboration with developers and platform engineers.
  • Set up, develop, and maintain the test automation repository, including its structure, standards, shared fixtures, and reporting.
  • Build and maintain automated REST API tests, covering functional, schema, negative, and regression scenarios.
  • Develop and maintain stable end-to-end UI tests using Playwright.
  • Test asynchronous processes, including agent deployments, environment provisioning, and WebSocket log streaming.
  • Perform integration and smoke testing of AI agents, including authentication, API endpoints, MCP tool calls, logs, traces, and memory persistence.
  • Integrate automated tests into CI/CD pipelines and implement quality gates, nightly regression runs, and post-deployment smoke tests.
  • Collaborate with development, AI, and platform teams to investigate issues and improve overall software quality.
  • 5+ years of experience in software quality assurance and test automation.
  • Proven experience defining test strategies for web platforms or SaaS products.
  • Experience setting up and owning test automation repositories using Python or TypeScript.
  • Strong hands-on experience with REST API test automation, including schema validation, asynchronous workflows, authentication, and access control.
  • Practical experience with Playwright and end-to-end testing of React or other SPA applications.
  • Experience integrating automated tests into GitLab CI or similar CI/CD tools, preferably with Docker.
  • Experience testing cloud-based applications and troubleshooting issues using logs, metrics, and traces.
  • Understanding of AI-based agents, including LLM behaviour, tools, memory, and non-deterministic outputs.
  • Daily experience using AI coding assistants such as GitHub Copilot or Claude Code, with the ability to critically review AI-generated code.
  • Strong analytical, problem-solving, and communication skills.

Nice to have

  • Experience with AWS services such as Lambda, API Gateway, DynamoDB, S3, CloudWatch, or Bedrock AgentCore.
  • Familiarity with AI agent frameworks such as LangGraph, AWS Strands, or CrewAI.
  • Knowledge of MCP and A2A protocols.
  • Experience with LLM evaluation tools such as DeepEval, Promptfoo, or Ragas.
  • Knowledge of vector databases and retrieval systems.
  • Experience with monitoring and observability tools.
  • Knowledge of OWASP Top 10 and security testing tools such as Burp Suite or OWASP ZAP.
  • Experience working in regulated or GxP environments.

Project details

  • Start: ASAP
  • Remote: remote
  • Location: Remote, PL
  • Rate: 120-130 EUR/h
  • Languages: English, Polish

Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing

Found on nofluffjobs.com

  1. Job boardNew

    IT Business Analyst

    Confidential· Warsaw, Poland· Hybrid· Senior

    • Requirements Engineering
    • SQL
    • Scrum
    • +6

  2. Job boardNew

    QA/Data Quality Engineer

    Primaris Services Sp. z o.o.· Warszawa, Warszawa, Poland· Remote· Senior· zł 110–125 / h

    • SQL
    • Python
    • Data Testing
    • +9

Browse all jobs
Sign up free to apply