Software Engineer - AI Evaluation & Automation

Project overview

Position: Software Engineer - AI Evaluation & Automation

Job Description:

JD s are attached. Feel free to reach for any questions.

Based on the customer feedback, the following skills and experience are must-haves:

Software Engineer AI Evaluation & Automation

Role Overview

Help build and scale the tooling we use to measure how well AI-powered software

development tools actually perform. You'll develop evaluation harnesses, automate

benchmark runs, and help make sure the results we produce are reproducible and hold up

to scrutiny. This is an engineering role, but a lot of the work is about getting the

measurement right, not just automating it.

Key Responsibilities

cases, including turning real engineering artifacts like merged pull requests into

repeatable benchmark tasks.

with reproducible run environments (pinned dependencies, containerized runs,

isolated worktrees) so results stay comparable over time.

consistent and correct rather than just repeatable.

measures, including cost and latency.

outcome, and find ways to make the workflows more reliable and more automated.

evaluations work and what they found for both technical and leadership audiences.

Required Skills & Experience

developer tooling, or test and validation systems.

the specific language background is flexible.

behave, and of containerization with Docker.

engineering workflows.

as why a judge can be consistent but still wrong, why a single run can mislead, and how

benchmark contamination happens.

valid, and analyze results carefully.

Code, Devin, or Cursor, and command of the best practices for working with them

eJectively.

Preferred Experience

execution-based grading that verifies against tests.

against human raters.

files to the tests that cover them.

evaluation datasets.

Job Facts: Contract Corp To Corp | Contract Independent | Contract W2 | On-site | Depends on Experience

Skills: LLM, Codex

Reference: Dice Id: 91082022 | Position Id: 9125023

• Coding Agent Evaluation: Experience benchmarking coding agents such as Claude Code, Codex, or similar tools against real repositories and engineering tasks.
• Evaluation Harness Development: Experience building Python-based frameworks to execute, test, and score AI agents in controlled, reproducible environments.
• Benchmark & Grading Design: Experience creating evaluation tasks, reference solutions, hidden tests, rubrics, and execution-based graders.
• Strong Software Engineering: Proficiency in Python, Git, CI/CD, automated testing, debugging, and working with complex, multi-file repositories.
• Reproducibility & Contamination Controls: Experience with sandboxed execution, repeatable evaluations, and preventing benchmark leakage or state contamination.
• Failure Analysis: Ability to analyze agent traces, identify root causes, and improve evaluation accuracy.
• Build and integrate evaluation harnesses and automation for software development use
• Build versioned, repeatable processes to evaluate AI tools, models, and harnesses,
• Validate and calibrate evaluation approaches against human judgment, so scores are
• Support execution-based benchmarking across quality, productivity, and eJiciency
• Analyze results across repeated runs, looking at variance, failure patterns, and cost per
• Work with engineering and data teams to improve the tooling, and document how the
• Strong software engineering background, with real experience building automation,
• Proficient in at least one general-purpose language such as Python, Java, or JavaScript
• Solid working knowledge of Git, including how branches, history, and working trees
• Experience with APIs, development environments, CI/CD pipelines, and typical
• Understanding of how AI, LLM, or agent evaluation works and where it goes wrong, such
• Able to troubleshoot technical problems, think clearly about whether a measurement is
• Hands-on experience using AI coding tools and agentic harnesses such as Claude
• Experience designing benchmarks or evaluations for software systems, especially
• Familiarity with LLM-as-judge or agent-as-judge approaches, and how to check them
• Experience with build-system-aware test selection, such as Bazel or mapping changed
• Experience building reproducible test environments and managing versioned

Project details

• Start: flexible
• Remote: onsite
• Location: San Diego, CA, US, US