Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team is representing a confidential AI technology client seeking Freelance Agent Evaluation Engineers in Saudi Arabia. This is a project-based opportunity focused on designing and testing evaluation frameworks for AI coding agents.

You will create challenging real-world developer tasks within simulated environments, define success criteria, write comprehensive tests, and iterate based on feedback. This role requires deep technical judgment to design evaluations that distinguish good solutions from bad ones while accepting all valid approaches.

Responsibilities

• Build realistic developer environments including codebases, infrastructure, documentation, and development history context
• Design intermediate-complexity tasks with clear prompts and solvability criteria for AI agent evaluation
• Write functional and integration tests that verify agent solutions while accepting diverse valid approaches
• Iterate on tasks and tests based on quality assurance feedback
• Analyze agent failures to refine evaluation robustness and fairness
• Complete tasks efficiently to maximize your effective compensation rate

What We're Looking For

• Master's degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, Computational Linguistics/NLP, Information Systems, or related field (Bachelor's degree accepted with 5+ years professional experience)
• Minimum 3 years professional experience in QA automation, testing, or cybersecurity roles
• 5+ years total software development experience
• Strong proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• English proficiency at B2+ level or higher
• Deep understanding of model limitations and edge cases that reveal solution quality differences

How We Work With You

Interested candidates should apply directly through Get A Job.ai with their CV in English, indicating their English proficiency level. Our recruiter will screen qualified applicants and submit selected profiles to our client. Please do not contact the client directly.

This is a freelance, project-based engagement that fits around your primary professional or academic commitments. Work is remote and paid per accepted task based on your qualification tier and completion efficiency.

Pay

Compensation is task-based with rates depending on your qualification tier and efficiency. Faster completion increases your effective hourly rate, with potential earnings equivalent to competitive freelance engineering rates.

Get A Job.ai is an equal opportunity recruiter. We welcome applications from all qualified candidates regardless of race, color, religion, sex, national origin, age, disability, or any other protected characteristic.