Other open roles
Applied Engineer, Evaluations Role
Work Expert (WE) is an independent publishing and referral website. We are not a recruiter, hiring manager, agent or employer, and we are not affiliated with or endorsed by Mercor. Applying takes you to the platform's own website, where we may be recorded as the referring source. We may receive a referral fee at no additional cost to you. Read our full affiliate disclosure →
About the Role
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more. We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models. In this role, you will:
What You'll Do
- Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs.
- Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions.
- Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts.
- Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria that unnecessarily constrain how a problem can be solved.
- Run evaluations repeatedly across multiple models and harnesses. Investigate whether failures come from the agent’s solution, the environment, or the grading, and establish that tasks expose meaningful weaknesses in agent performance.
- Refine evaluations through repeated testing and review. Document findings and grading decisions, and work through feedback.
- Strong programming fundamentals and practical experience working in substantial and complex codebases. Languages we create evals for include TypeScript/JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++ and Rust
- The ability to understand unfamiliar code, investigate subtle behavior, and assess whether different implementations solve the same problem correctly.
You're a Good Fit If You
Role Highlights
Pay & Payout
Independent contractor engagement. Apply via the link below - Mercor handles onboarding and pays weekly via Stripe or Wise.
Common Questions
Opens Mercor in a new tab. You pay nothing; we may earn a referral fee.
Get similar jobs every week
Similar roles
More coding work you may qualify for.
Audio Engineer (Speech / TTS Audio Specialist) – Brazilian Portuguese
We're looking for audio engineers with strong experience editing and quality-checking voice recordings for speech and text-to-speech (TTS) applications, specifically for Brazilian
Infrastructure / Site Reliability Engineer (SRE)
Mercor connects exceptional technical talent with leading organisations working on ambitious technology and AI initiatives. We are looking for experienced Infrastructure / Site
CAD / PCB / 3D Design Engineer — AI Evaluation
Mercor is partnering with a leading AI lab to measure how well frontier AI models perform real engineering design work. We're hiring experienced CAD, PCB, and 3D design engineers