Quality Systems Lead

About This Opportunity

We are representing a confidential AI platform company in their search for a Quality Systems Lead based in London. Our client is a well-funded organisation at the frontier of enterprise AI, delivering curated training data to physical AI companies, frontier labs, and regulated industries. This is a rare hybrid role combining quality engineering with operational leadership—you will build automated evaluation systems and simultaneously manage a distributed audit team that produces the ground truth those systems learn from.

Responsibilities

• Design and build automated dataset quality evaluation pipelines using model-assisted screening, LLM-as-judge frameworks, agreement analysis, anomaly detection, and drift monitoring across diverse data types
• Hire, train, calibrate, and manage a dedicated audit team of approximately ten specialists based in India, measuring performance by inter-rater agreement and defect catch rate rather than volume
• Convert audit outputs into labelled ground truth that trains and validates the automated evaluation layer, deliberately managing the ratio of automated to manual coverage as the function scales
• Own the quality standard for each data modality delivered—written rubrics with edge cases, golden datasets, and acceptance criteria aligned with customer requirements before production begins
• Build scoring systems that rank annotator and reviewer performance to inform routing, staffing, and offboarding decisions
• Set certification pass thresholds ensuring no individual works a project queue without demonstrating competency against the standard
• Report quality KPIs to leadership and into customer-facing reporting: accuracy against specification, inter-annotator agreement, rework rate, cost of rework, and coverage metrics
• Collaborate with project management on remediation—you diagnose and measure, production executes fixes, and the standard remains independent of those being evaluated
• Own the unit economics of quality: cost per audited unit and the coverage achieved per budget spent
• Partner with product and engineering teams to migrate internal quality measurement capabilities into native platform features

What We're Looking For

Core Requirements:

• 4+ years owning both technical and operational outcomes in a data, AI, or service delivery environment where quality was rigorously measured
• Hands-on proficiency in Python and SQL—you build the analysis and tooling yourself rather than delegating implementation
• Practical experience applying models to quality or evaluation problems: LLM-as-judge, model-assisted QA, automated evaluation, anomaly detection, or classifier-based screening
• Demonstrated expertise in sampling methodology and agreement statistics (Cohen's kappa, Fleiss' kappa, F1 against ground truth) applied to production data at scale
• Experience hiring, training, and managing a team—ideally an audit, review, or QA function, and ideally distributed across geographies
• Track record of building a quality framework or function from the ground up, including the reporting that leadership and customers rely on

Who Succeeds in This Role:

• You build and you operate—you will write the evaluation pipeline and also run weekly calibration sessions with auditors in another country
• Your instinct on a coverage problem is to automate it: you reach for a model, heuristic, or better sampling design before adding headcount
• Statistically literate in a practical way—you understand sampling design, agreement statistics, and can spot when a metric is being gamed rather than measured
• You can hold a calibrated standard across a distributed team you don't sit with—you understand that ten uncalibrated auditors produce ten different standards
• Strong writer—much of this role is producing rubrics a distributed workforce can follow without you in the room
• You hold a standard under commercial pressure and bring the evidence that makes it stick with delivery teams and clients
• Systems thinker—as interested in why a failure recurs across projects as in any single project's defect rate

Bonus Experience:

• Direct experience with annotation, evaluation, or model-training workflows, and understanding of what frontier AI labs accept as quality evidence
• STEM degree or background in data science or research engineering
• Multilingual delivery and linguistic quality assessment

How We Work With You

Candidates apply directly through Get A Job.ai. Our talent team will conduct an initial screening to understand your background and confirm alignment with the role. Qualified candidates are then submitted to our client for consideration. Please do not attempt to contact the client directly—all communication and coordination will be managed through our recruiting team to ensure a professional and structured process.

Pay

Compensation details will be discussed during the screening process based on experience and qualifications.

Equal Opportunity: Get A Job.ai is committed to inclusive hiring practices and equal opportunity employment. We welcome applications from candidates of all backgrounds.