Freelance Agent Evaluation Engineer

About This Opportunity

We are representing a confidential AI technology organization seeking Freelance Agent Evaluation Engineers based in Japan. This is a project-based engagement focused on building evaluation datasets for AI coding systems.

Our client is developing advanced testing environments to assess how well AI models handle real-world developer workflows. You will design challenging tasks within realistic simulated development environments, create evaluation criteria, and write tests that fairly judge AI-generated solutions.

Responsibilities

• Build realistic developer environments including virtual company codebases, infrastructure, and contextual materials (tickets, documentation, conversation history)
• Design intermediate-state tasks with clear prompts and success criteria that challenge AI agents
• Write comprehensive tests that accept all valid solution approaches while rejecting incorrect ones
• Iterate on tasks and tests based on quality assurance feedback
• Analyze agent solution failures and refine evaluation frameworks to ensure fairness and robustness

What We're Looking For

Required Experience:

• 5+ years in software development
• Minimum 3 years in QA automation/testing or cybersecurity roles
• Strong proficiency with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Proven experience writing functional and integration tests
• English proficiency at B2+ level or higher

Education:

• Master's degree in Computer Science, Software Engineering, Data Science, AI/Machine Learning, NLP, Information Systems, or related field
• Bachelor's degree acceptable with 5 years of field experience

Key Skills:

• Deep understanding of where AI models fail in coding tasks
• Ability to design scenarios that differentiate quality solutions
• Expertise in creating flexible test frameworks that handle multiple valid approaches

How We Work With You

Candidates apply directly through Get A Job.ai. Our talent team will conduct an initial screening to discuss your background and the project scope. We then coordinate qualifications and submit qualified candidates to our client for final placement.

Please do not attempt to contact the client organization directly, as all applications must be processed through our recruiting team to be considered.

Please submit your CV in English and clearly indicate your level of English proficiency.

Pay

This is a project-based, freelance engagement. Compensation is paid per accepted task, with rates dependent on qualification tier achieved and task completion efficiency. The effective rate can reach up to $50 USD per hour equivalent based on your pace and performance.

This is a part-time, remote opportunity designed to fit around your primary professional or academic commitments.

Equal Opportunity: Get A Job.ai is committed to fostering an inclusive recruitment process. We welcome applications from candidates of all backgrounds and work with our clients to ensure fair consideration.