Shyft6
This is a remote position.
Reporting to Manager, Quality Engineering & AI Validation, focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. Builds benchmark scenarios, defines scoring rubrics, evaluates business usefulness, and identifies failure patterns that conventional pass or fail software testing would not catch.
Key Responsibilities
AI Output Evaluation
- Design and execute structured evaluations for AI-enabled features and workflows.
- Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk.
- Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations.
Benchmark & Rubric Development
- Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards.
- Develop rubrics that allow quality to be measured consistently across releases and changes.
Workflow & Model Change Validation
- Compare performance across prompt versions, workflow revisions, tools, and models.
- Support release decisions with evidence on quality regression or improvement.
Business & Domain Partnership
- Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts.
- Help define human-review thresholds and escalation patterns for higher-risk use cases.
Production Feedback
- Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time.
Requirements
Required Qualifications
- 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles.
- Strong writing, analysis, and pattern-recognition skills.
- Experience evaluating outputs against nuanced criteria rather than only binary correctness.
- Ability to work with structured rubrics, scenario libraries, and evidence-based reviews.
- Comfort collaborating across Engineering and business teams.
- Experience with finance, accounting, FP&A, transaction services, or business process design preferred.
·Bachelor’s degree preferred.
You Are
- Thoughtful, precise, and highly discerning.
- Strong at spotting subtle output problems others miss.
- Comfortable with ambiguity but disciplined in scoring and documentation.
- Focused on trust, usefulness, and business reality.
Benefits
Originally posted on Himalayas
To apply for this job please visit himalayas.app.
Working in United States
The United States of America (USA), also known as the United States (U.S.) or America, is a country primarily located in North America. It is a federal republic consisting of 50 states and a federal capital district, Washington, D.C. The 48 contiguous states border Canada to the north and Mexico to the south, with the semi-exclave of Alaska in the northwest and the archipelago of Hawaii in the Pacific Ocean. The United States also asserts sovereignty over five major island territories and various uninhabited islands in Oceania and the Caribbean. It is a megadiverse country, with the world's th
More jobs at Shyft6
Keep exploring on Get A Job.ai
Not quite the right fit? Your next opportunity is a click away.
- Browse all jobs
- More jobs by category
- Remote jobs you can do from anywhere
- Research typical pay for this role
- Set a job alert so new matches reach you first
- Upload your resume to apply faster
Hiring instead? Post a job and reach candidates searching right now.