Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team at Get A Job.ai is representing a confidential AI technology client seeking Freelance Agent Evaluation Engineers in Estonia. This is a project-based engagement focused on testing and evaluating AI coding systems through the creation of challenging development scenarios and assessment criteria.

Please note: applications must include a CV in English with an indication of English proficiency level.

Responsibilities

• Build realistic simulated developer environments including codebases, infrastructure, documentation, and development history context
• Design technical tasks from intermediate states of these environments, crafting prompts and defining solution criteria for AI agent evaluation
• Write comprehensive tests that verify agent solutions, accepting all valid approaches while rejecting incorrect implementations
• Iterate on tasks and test suites based on QA feedback, analyzing agent failures and refining evaluation frameworks to ensure fairness and robustness
• Identify scenarios that reveal performance differences between AI model capabilities

What We're Looking For

• 5+ years of professional software development experience
• Strong proficiency with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Demonstrated experience writing functional and integration tests
• English proficiency at B2 level or higher
• Master's degree in Computer Science, Software Engineering, Data Science, AI/ML, Computational Linguistics/NLP, Information Systems, or related field; OR Bachelor's degree with 5+ years of relevant experience
• Minimum 3 years of professional experience in QA automation, testing, or cybersecurity roles
• Understanding of where AI models struggle and ability to design scenarios that challenge frontier capabilities

How We Work With You

Candidates apply directly through Get A Job.ai. Our recruiting team conducts an initial screening and qualification assessment. Once qualified, we submit your profile to our client for consideration. Please do not contact the client directly—all communication flows through our team to protect both your interests and the client relationship.

This is a remote, project-based freelance engagement that can fit around other professional or academic commitments. Work involves completing discrete tasks with payment per accepted deliverable. Compensation is structured by qualification tier achieved, with efficient completion improving effective hourly rates.

Pay

Compensation details will be discussed during the screening process based on qualification tier and project scope.

Equal Opportunity: Get A Job.ai is committed to inclusive recruiting practices. We welcome applications from candidates of all backgrounds and work to ensure fair consideration throughout our process.