Freelance Agent Evaluation Engineer

About This Opportunity

Our talent team is representing a confidential AI technology client seeking Freelance Agent Evaluation Engineers in South Africa. This is a project-based engagement focused on building and evaluating datasets that test how well AI coding agents handle real-world developer challenges.

You will design challenging tasks within realistic simulated development environments, create evaluation criteria, write comprehensive tests, and iterate based on feedback. This role requires deep technical understanding of where AI models struggle and how to create fair, robust evaluations.

Responsibilities

• Build realistic developer environments including virtual company codebases, infrastructure, and authentic context (tickets, documentation, team conversations)
• Design intermediate-state tasks with clear prompts and success criteria that AI agents can attempt to solve
• Write functional and integration tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones
• Iterate on tasks and tests based on quality assurance feedback, analyzing agent failures and refining evaluations
• Ensure evaluation criteria are neither too strict nor too lenient, balancing comprehensive coverage with fairness

What We're Looking For

• 5+ years of professional software development experience
• Strong proficiency with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Demonstrated experience writing functional and integration tests
• Master's Degree in Computer Science, Software Engineering, Data Science, AI/ML, NLP, Information Systems, or related field (Bachelor's degree accepted with 5+ years field experience)
• Minimum 3 years professional experience in QA automation, testing, or cybersecurity roles
• English proficiency at B2+ level or higher
• Deep understanding of where AI models fail and what scenarios reveal solution quality differences

How We Work With You

Candidates apply directly through Get A Job.ai. Our recruiting team conducts initial screening conversations to understand your technical background and project fit. Once qualified, we submit your profile to our client for consideration. Please do not contact the client directly—all communication flows through our team to ensure a smooth process.

This is a project-based freelance engagement, not permanent employment. Work is remote and part-time, designed to fit around your primary professional or academic commitments. Compensation is task-based, with your rate depending on qualification tier and task completion efficiency.

Pay

Payment structure is per accepted task. Rates scale based on your qualification tier and completion efficiency, with potential effective hourly rates up to the equivalent of $40 USD per hour for faster-paced work.

Get A Job.ai is an equal opportunity recruiter. We welcome applications from all qualified candidates regardless of race, color, religion, sex, national origin, age, disability, or any other protected characteristic.