Expert Consultant AI Testing / Senior QA – Agent Architectures

Project overview

Within a large group committed to transforming and industrializing its delivery processes, the consultant will work on projects related to the integration of artificial intelligence and the development of solutions based on agent architectures.

The mission focuses on two main areas:

In this context, the objective is to define and industrialize a true quality and testing approach adapted to artificial intelligence systems, and more specifically to agent architectures.

The consultant will be involved in the quality of the development cycle, the evaluation of AI agents, and the security of the final product.

1.1. Scoping Quality and Defining Test Strategy

1.2. Designing and Executing AI Tests

1.3. Integrating Quality into SDLC and CI/CD Processes

1.4. Industrialization and Observability

1.5. Supporting Teams

Profile Sought

We are looking for a Senior Test Consultant / QA Expert with solid experience in software test strategy and concrete expertise in evaluating artificial intelligence solutions, particularly LLMs and agent architectures.

The candidate must be able to combine a global vision of software quality with a technical and operational approach to testing AI systems.

They should also be comfortable in developing environments involving multiple teams and requiring the structuring of practices and responsibilities.

2.1. Essential Experience

2.2. Technical Skills

Software Testing and QA

AI / LLM / Agent Architectures Testing

Delivery and DevSecOps

Development and Scripting

• Setting up an agentic SDLC: structuring and securing an end-to-end development cycle integrating AI agents, from requirements to deployment, with quality controls at each stage.
• Developing a business prototype integrating AI: qualifying an agent solution to enhance an existing business process, particularly for analyzing incoming documents and leveraging information in an operational process.
• Main Missions
• Define and structure the test strategy adapted to solutions integrating AI components and agent architectures.
• Identify quality requirements and acceptance criteria at each stage of the development cycle.
• Define test activities, quality controls, and performance indicators.
• Translate business needs and expected behaviors into measurable validation criteria.
• Define Go/No Go decision criteria for production releases.
• Design and execute test scenarios for LLMs, AI agents, and multi-agent orchestrations.
• Evaluate the conformity of agent responses and behaviors against business expectations.
• Test the robustness of solutions against various situations and unexpected inputs.
• Identify and measure risks of hallucination, out-of-scope responses, and non-compliant behaviors.
• Evaluate interactions between agents, reviewers, and external tools.
• Define reproducible test sets and evaluation metrics.
• Qualify AI components and their integration into end-to-end functional flows.
• Integrate test and quality control activities throughout the software development cycle.
• Define and implement Quality Gates at various delivery stages.
• Ensure traceability of requirements, tests, and obtained results.
• Participate in the integration of automated tests into CI/CD pipelines.
• Contribute to defining controls associated with User Stories, Pull Requests, and deployments.
• Implement mechanisms for reporting results and monitoring quality.
• Automate test and AI solution evaluation mechanisms.
• Implement monitoring and observability mechanisms for agents.
• Monitor execution traces, tool calls, errors, latencies, and costs.
• Structure and maintain test datasets and scenarios.
• Contribute to securing agent architectures, particularly against prompt injection risks, data exfiltration, and unauthorized actions.
• Formalize results and produce evaluation reports usable by the teams.
• Support technical and business teams in adopting AI testing practices.
• Clarify the role and responsibilities of the QA function in agent projects.
• Harmonize testing practices and quality criteria across different teams.
• Raise awareness among teams about specific risks related to LLMs and multi-agent architectures.
• Be proactive in the continuous improvement of quality and delivery practices.
• Significant experience in defining and managing test strategies on mature SDLCs.
• Mastery of different testing approaches: functional, technical, automated, and exploratory.
• Concrete experience in deploying software solutions in demanding environments.
• Good command of CI/CD environments, code reviews, and Quality Gates.
• Proven experience in evaluating solutions based on LLMs, AI agents, or multi-agent orchestrations.
• Ability to translate business requirements into test scenarios and measurable validation criteria.
• Experience in transformation contexts, with multiple teams and evolving organizations.
• Expertise in test strategy and quality assurance.
• Mastery of functional, technical, automated, and exploratory testing.
• Definition of test plans, scenarios, and acceptance criteria.
• In-depth knowledge of LLM evaluation methods.
• Experience in testing and qualifying AI agents.
• Mastery of hallucination, robustness, and response conformity issues.
• Understanding of agent architectures and multi-agent orchestrations.
• Knowledge of security challenges specific to AI systems.
• Mastery of CI/CD practices and deployment processes.
• Experience in implementing Quality Gates.
• Good knowledge of Git workflows, Pull Requests, and code reviews.
• Ability to integrate automated tests into delivery pipelines.
• Ability to develop test scripts and automations.
• Good command of Python and associated testing frameworks.
• Ability to leverage APIs and execution traces to qualify systems.

Project details

• Start: ASAP
• Duration: 1 an renouvelable
• Remote: hybrid
• Location: Paris, Île-de-France, FR
• Rate: 100-410 EUR/h
• Languages: French