Agent Evaluation Engineer (Freelance)

FREELANCE AGENT EVALUATION ENGINEER

The Opportunity

Mindrift is seeking a Agent Evaluation Engineer (Freelance) to join our team in Canada (nationwide) (nationwide). We are focused on testing, evaluating, and improving AI systems by building datasets that challenge advanced AI coding agents. This role involves creating realistic developer environments, designing tasks within these environments, writing tests for agent solutions, and iterating based on feedback.

Day-to-Day Responsibilities

• Build realistic developer environments with a virtual company's codebase, infrastructure, and context (tickets, docs, conversations)
• Create challenging tasks from intermediate states of these environments, crafting the prompt, defining "solved," and ensuring solvability by an AI agent
• Write tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones without being too strict or lenient
• Iterate on tasks and tests based on quality assurance feedback, reviewing agent solutions, analyzing failures, and refining until the evaluation is fair and robust

Requirements

• 5+ years in software development
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
• Experience writing tests (functional, integration)
• English proficiency - B2+

Working at Mindrift

At Mindrift, we connect specialists with project-based AI opportunities for leading tech companies. Participation is not permanent employment but involves building a dataset to evaluate AI coding agents' performance in real-world developer tasks.

Apply Today

To apply, complete your application directly on this page, or you'll be redirected to the employer's application platform to finish submitting there.