Freelance Agent Evaluation Engineer

About This Opportunity

We are representing a confidential AI technology organization that is building evaluation datasets for next-generation coding agents. This is a freelance, project-based role for experienced software engineers who want to shape how AI systems handle real-world development tasks.

You will design challenging scenarios within realistic simulated development environments, define evaluation criteria, and write tests that fairly assess AI agent performance. This work goes far beyond data labeling—you're architecting complex technical puzzles that reveal the true capabilities of frontier models.

Responsibilities

• Build realistic developer environments including codebases, infrastructure, tickets, documentation, and conversation history that form believable development contexts
• Design intermediate-state tasks with clear prompts and solution definitions that are solvable by AI agents
• Write robust tests that accept all valid solution approaches while rejecting incorrect ones—neither overly strict nor lenient
• Iterate on tasks and tests based on QA feedback, analyzing agent failures and refining evaluations until they are fair and production-ready
• Understand where advanced models fail and craft scenarios that reveal meaningful performance differences

What We're Looking For

• 5+ years in software development
• Core technical stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
• Testing expertise: functional and integration testing experience required
• Education: Master's degree in Computer Science, Software Engineering, Data Science, AI/ML, NLP, Information Systems or related field (Bachelor's accepted with 5+ years experience)
• Professional background: Minimum 3 years in QA automation, testing, or cybersecurity roles
• English proficiency: B2+ level required; please submit your CV in English and indicate your proficiency level
• Location: Ireland (remote/freelance)

Why This Is Challenging: Frontier AI models are already strong at coding. Creating tasks that genuinely test the best systems requires deep understanding of model limitations. Writing tests that accept all correct solutions while rejecting flawed ones is significantly harder than traditional test development.

How We Work With You

Candidates apply through Get A Job.ai. Our talent team will conduct an initial screening and qualification assessment. Once qualified, we submit your profile to our client. Please do not contact the client directly. After placement, you will complete project-based tasks on a flexible schedule that fits around your primary commitments.

Work structure: Apply → Pass qualification → Join project → Complete tasks → Get paid per accepted task

Pay

This is paid per accepted task, not hourly. Your rate depends on the qualification tier you achieve. The client indicates compensation can reach the equivalent of $50/hour for efficient completion. Faster, high-quality work increases your effective hourly rate. Payment is processed after task acceptance.

Equal Opportunity

Get A Job.ai is committed to equal employment opportunity. We evaluate candidates based on qualifications and do not discriminate on the basis of race, color, religion, sex, national origin, age, disability, or any other protected characteristic.