Work Expert (WE) is an independent publishing and referral website. We are not a recruiter, hiring manager, agent or employer, and we are not affiliated with or endorsed by Mercor. Applying takes you to the platform's own website, where we may be recorded as the referring source. We may receive a referral fee at no additional cost to you. Read our full affiliate disclosure →
About the Role
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
What You'll Do
You're a Good Fit If You
3+ years professional software engineering
Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Familiarity with SWE-Bench (Verified) or similar repository benchmarks
Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
Independent contractor engagement. Apply via the link below - Mercor handles onboarding and pays weekly via Stripe or Wise.
Common Questions
Mercor has a reliability rating of very reliable based on the listings we track, with an onboarding time of 1-3 weeks (interview + assessment + trial). See our full Mercor review for the details behind that rating.
Applying takes you to Mercor's own website, where the application is completed. We are not a recruiter or employer and do not process applications directly - we may be recorded as the referring source.
$70-$90/h - Independent contractor engagement. Apply via the link below - Mercor handles onboarding and pays weekly via Stripe or Wise.
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
