Freelance Agent Evaluation Engineer

About This Opportunity

We are representing a confidential AI technology organization that is building evaluation datasets for next-generation coding agents. This is a project-based freelance role where you will design realistic developer environments and challenging tasks that test how well AI models handle real-world software engineering work.

This is not permanent employment. You will work on a project basis, submitting tasks and getting paid per accepted deliverable. Participation is remote and designed to fit around your primary professional or academic commitments.

Responsibilities

• Build realistic simulated developer environments including virtual company codebases, infrastructure, tickets, documentation, and conversation histories that form believable development contexts
• Design challenging intermediate-state tasks within these environments, crafting prompts and defining solution criteria that an AI agent can attempt to solve
• Write comprehensive tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones without being overly strict or lenient
• Iterate on tasks and tests based on QA feedback, reviewing agent solutions, analyzing failure points, and refining evaluation criteria until assessments are fair and robust
• Create scenarios that reveal meaningful differences between good and bad AI-generated solutions, targeting edge cases where frontier models struggle

What We're Looking For

• 5+ years of professional software development experience
• Strong proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Hands-on experience writing functional and integration tests
• A Master's degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, NLP, Information Systems, or related fields (Bachelor's degree accepted with 5+ years of experience)
• Minimum 3 years of professional experience in QA automation, testing, or cybersecurity roles
• English proficiency at B2 level or higher
• Deep understanding of where AI models struggle and what scenarios expose the difference between correct and incorrect solutions

Please submit your CV in English and indicate your level of English proficiency.

How We Work With You

Candidates apply directly through Get A Job.ai. Our talent team will review your application and conduct an initial screening. If you're a strong match, we will facilitate qualification assessments. Once qualified, we submit you to our client for project participation. Please do not contact the client directly — all communication flows through our recruiting team to protect both your candidacy and the client relationship.

The process: Apply → Pass qualifications → Join project → Complete tasks → Get paid per accepted task.

Pay

Compensation is paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks. Faster, high-quality work raises your effective hourly rate, with top performers reaching the equivalent of $50/hour USD.

Get A Job.ai is an equal opportunity recruiter. We welcome applications from candidates of all backgrounds and do not discriminate on the basis of race, color, religion, sex, national origin, age, disability, or any other protected characteristic.