Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team at Get A Job.ai is representing a confidential AI technology organization on a unique project-based engagement. We're seeking experienced software engineers to help evaluate and improve AI coding agents through creating challenging, realistic development scenarios.

This is a freelance, project-based role—not permanent employment. You'll work remotely on your own schedule, designing evaluation frameworks that test how well AI models handle real-world developer tasks.

Responsibilities

• Build realistic virtual development environments including codebases, infrastructure, documentation, and contextual elements that simulate authentic development workflows
• Design intermediate-state tasks with clear prompts and success criteria that are solvable by AI agents
• Write comprehensive tests that verify agent solutions, accepting all valid approaches while rejecting incorrect implementations
• Iterate on tasks and tests based on quality assurance feedback, analyzing agent failures and refining evaluations until they are fair and robust
• Create scenarios that genuinely challenge frontier AI models and reveal meaningful differences between quality solutions

What We're Looking For

Required Experience:

• 5+ years in software development
• 3+ years of professional experience in QA automation, testing, or cybersecurity roles
• Strong proficiency with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• English proficiency at B2 level or higher

Educational Requirements:

• Master's degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, Computational Linguistics, Information Systems, or related fields
• Bachelor's degree accepted with 5+ years of relevant experience

Important Clarifications:

• This is not data labeling or prompt engineering work
• You won't write code from scratch—you guide and evaluate AI-generated solutions
• Tasks must genuinely challenge advanced AI models and accommodate multiple valid solution approaches

How We Work With You

Candidates apply directly through Get A Job.ai. Our recruiting team conducts an initial screening to understand your background and technical expertise. We then submit qualified candidates to our client for their review and qualification process. Once you pass the client's qualifications, you'll join the project and begin completing tasks.

Please do not contact the client directly. All communications and applications must go through Get A Job.ai to be considered.

Pay

Compensation is paid per accepted task. Your rate depends on the qualification tier you achieve and your efficiency in completing tasks. The client indicates rates can reach the equivalent of up to $50 USD per hour for top performers. Because payment is task-based rather than hourly, working at a faster pace increases your effective hourly rate.

This is a flexible, part-time opportunity designed to fit around your primary professional or academic commitments.

Equal Opportunity

Get A Job.ai is committed to providing equal employment opportunities to all applicants regardless of race, color, religion, sex, national origin, age, disability, or any other protected characteristic. We evaluate all candidates based on qualifications and professional merit.