Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team at Get A Job.ai is representing a confidential AI technology company seeking Freelance Agent Evaluation Engineers in Spain. This is a project-based opportunity focused on evaluating and improving AI coding systems through realistic scenario development and rigorous testing.

You'll design challenging evaluation environments that test how well AI models handle real-world developer tasks. This involves creating believable development contexts—complete with codebases, infrastructure, documentation, and workflows—then crafting tasks that reveal the true capabilities and limitations of frontier AI systems.

Responsibilities

• Build realistic simulated developer environments with credible development histories, including tickets, documentation, and team conversations
• Design intermediate-state tasks with clear prompts and success criteria that AI agents must solve
• Write comprehensive tests that accept all valid solution approaches while rejecting incorrect ones
• Iterate on tasks and evaluation criteria based on QA feedback and agent performance analysis
• Review agent solutions, identify failure patterns, and refine evaluations until they are fair and robust

What We're Looking For

• 5+ years in software development
• Strong expertise in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• Master's degree in Computer Science, Software Engineering, Data Science, AI/ML, NLP, Information Systems, or related field (Bachelor's accepted with 5+ years experience)
• Minimum 3 years professional experience in QA automation, testing, or cybersecurity roles
• English proficiency at B2+ level
• Deep understanding of where AI models struggle and what scenarios differentiate strong from weak solutions

This is not: data labeling, basic prompt engineering, or writing code from scratch. The AI agent writes the code—you design the challenges and evaluate the results.

How We Work With You

When you apply through Get A Job.ai, our recruiters will screen your qualifications and facilitate your connection with our client. You'll complete qualification assessments, join project work, and submit completed tasks for payment. Please do not contact the client directly—all communication flows through our recruiting team to ensure proper coordination.

This is remote, project-based freelance work that fits around your primary commitments. Compensation is paid per accepted task, with rates determined by your qualification tier and task completion efficiency.

Pay

Compensation structure: paid per accepted task based on qualification tier, with effective rates reaching the equivalent of $40/hour for efficient completion. Your CV and application are required in English.

Equal Opportunity

Get A Job.ai is committed to providing equal employment opportunities to all applicants regardless of race, color, religion, sex, national origin, age, disability, or any other protected characteristic.