Freelance Agent Evaluation Engineer

About This Opportunity

We are representing a confidential AI technology organization seeking Freelance Agent Evaluation Engineers for a project-based engagement. This is a remote opportunity based in Argentina focused on evaluating and improving AI coding systems through the creation of realistic development scenarios and robust testing frameworks.

This is not permanent employment — it is project-based freelance work that can fit around your primary professional or academic commitments.

Responsibilities

• Build realistic developer environments including virtual company codebases, infrastructure, and contextual elements such as tickets, documentation, and conversation histories
• Design challenging tasks from intermediate states of these environments, crafting prompts and defining clear success criteria that AI agents can solve
• Write comprehensive tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones with appropriate strictness
• Iterate on tasks and tests based on quality assurance feedback, reviewing agent solutions and analyzing failures to refine evaluations until they are fair and robust
• Create scenarios that genuinely challenge frontier AI models by understanding where models fail and what reveals differences between good and bad solutions

What We're Looking For

Required Experience:

• 5+ years in software development
• Minimum 3 years of professional experience in QA automation, testing, or cybersecurity roles
• Strong proficiency with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• English proficiency at B2+ level or higher

Education:

• Master's Degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, Computational Linguistics/NLP, Information Systems, or related fields; OR
• Bachelor's degree with 5+ years of relevant field experience

Important distinctions: This role is not data labeling or prompt engineering. You will not be writing code from scratch — the AI agent writes most code while you guide and evaluate the quality of solutions.

How We Work With You

Candidates apply directly through Get A Job.ai. Our talent team will conduct an initial screening, and qualified candidates will complete qualification assessments. Once approved, we submit your profile to our client for project assignment. Please do not contact the client directly.

The workflow is: Apply → Pass qualifications → Join project → Complete tasks → Get paid per accepted task.

Please submit your CV in English and clearly indicate your English proficiency level.

Pay

Compensation is paid per accepted task. Your rate depends on the qualification tier you achieve and task completion efficiency. Because payment is task-based rather than hourly, faster completion increases your effective hourly rate.

Equal Opportunity

Get A Job.ai is committed to fostering an inclusive recruiting process. We welcome applications from candidates of all backgrounds and provide equal consideration regardless of race, color, religion, gender, national origin, age, disability, or veteran status.