Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team at Get A Job.ai is representing a confidential AI technology client seeking a Freelance Agent Evaluation Engineer based in France. This is a project-based freelance opportunity focused on evaluating and improving AI coding capabilities.

In this role, you'll design challenging evaluation scenarios for AI coding systems. You'll build realistic developer environments—complete with codebases, infrastructure, and authentic context—then create tasks that test how well AI agents handle real-world development challenges. Your work will directly influence how next-generation AI models approach software engineering problems.

Responsibilities

• Build realistic simulated developer environments with believable codebases, infrastructure, tickets, documentation, and conversation histories
• Design evaluation tasks from intermediate states of these environments, crafting clear prompts and defining solution criteria
• Write robust tests that verify AI agent solutions—accepting all valid approaches while rejecting incorrect ones
• Iterate on tasks and tests based on quality assurance feedback, analyzing agent failures and refining evaluations until they're fair and challenging
• Ensure evaluations genuinely challenge advanced AI models by understanding where current systems fail

What We're Looking For

• 5+ years of professional software development experience
• Strong expertise in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• Master's degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, NLP, Information Systems, or related field (Bachelor's accepted with 5+ years experience)
• Minimum 3 years in QA automation, testing, or cybersecurity roles
• English proficiency at B2+ level or higher
• Deep understanding of software engineering principles and where AI models struggle with complex coding tasks

How We Work With You

Candidates apply directly through Get A Job.ai. Our recruiters will review your qualifications and conduct an initial screening. If you're a strong match, we'll submit your profile to our client for consideration. Please do not contact the client directly—all communication flows through our team to ensure a smooth process.

This is a project-based freelance engagement, not permanent employment. Work is remote and designed to fit around your primary professional or academic commitments. You'll complete a qualification process, join the project, and get paid per accepted task.

Pay

Compensation is task-based. Your rate depends on your qualification tier and task completion efficiency, with potential equivalent rates up to $50 USD per hour. Faster, high-quality work increases your effective hourly rate. Payment details will be discussed during the screening process.

Please submit your CV in English and clearly indicate your English proficiency level.

Get A Job.ai is committed to equal employment opportunity. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or other legally protected characteristics.