Other open roles
Engineering Manager – LLM Evaluation
Work Expert (WE) is an independent publishing and referral website. We are not a recruiter, hiring manager, agent or employer, and we are not affiliated with or endorsed by Turing. Applying takes you to the platform's own website, where we may be recorded as the referring source. We may receive a referral fee at no additional cost to you. Read our full affiliate disclosure →
Apply within 30 days of clicking - Turing only credits applications inside its 30-day attribution window.
About the Role
We are looking for an Engineering Manager to lead teams creating high-quality conversational data for training and evaluating advanced Large Language Models (LLMs). The role involves managing the design of realistic, multi-turn interactions between users and AI assistants, including tool-based workflows involving calendars, email, maps, cloud storage, and other applications. The manager will oversee project execution, maintain quality standards, guide technical decision-making, and collaborate with engineering, operations, and review teams to deliver reliable training and evaluation datasets.
What You'll Do
- Lead and manage teams responsible for creating and evaluating multi-turn LLM conversations.
- Define workflows, priorities, quality standards, and delivery goals for conversation data projects.
- Guide the design of realistic assistant interactions involving APIs, JSON, function calls, and external tools.
- Review complex scenarios involving decision-making, clarification, error handling, constraints, and task completion.
- Ensure conversations accurately reflect realistic user expectations and assistant capabilities.
- Establish quality-control processes and monitor team performance, consistency, and deliverable quality.
- Coach team members on technical reasoning, conversation design, tool usage, and adherence to project guidelines.
- Collaborate with engineering, product, operations, and quality teams to resolve issues and improve workflows.
You're a Good Fit If You
- Required skills: Software Development, API Integrations , JSON
- 5+ years of professional experience in software engineering, technical operations, data, AI, or a related technical field.
- Prior experience managing, mentoring, or leading technical teams.
- Strong understanding of APIs, JSON, data formats, logical reasoning, and structured workflows.
- Ability to guide teams in breaking down complex tasks into realistic conversational and tool-use flows.
- Strong understanding of assistant behavior, including tool selection, clarification, limitations, and error handling.
- Excellent written and verbal English communication skills.
- Strong attention to detail and ability to establish and enforce quality standards.
- Proven ability to manage multiple priorities, provide actionable feedback, and deliver projects within deadlines.
Role Highlights
Pay & Payout
Independent contractor engagement through Turing. Apply via the link below - Turing handles vetting, onboarding and payment.
Common Questions
Opens Turing in a new tab. You pay nothing; we may earn a referral fee.
Get similar jobs every week
Similar roles
More coding work you may qualify for.
DevOps and Cloud Infrastructure Engineer
Build and manage secure cloud infrastructure for PII detection workflows.
ML and Data Engineer — Data Quality & PII Compliance
Improve data quality and compliance for enterprise data pipelines.
ML and Data Engineer — Data Quality & PII Compliance
Validate data quality and compliance for enterprise data pipelines.