Freelance Agent Evaluation Engineer

About This Opportunity

We are representing a confidential AI technology organization that is building evaluation frameworks for next-generation coding agents. This is a project-based freelance engagement focused on creating challenging test scenarios that push the boundaries of what AI systems can accomplish in real-world development contexts.

You will design realistic developer environments—complete with codebases, infrastructure, tickets, documentation, and conversational context—then craft evaluation tasks that differentiate truly capable AI agents from those that merely appear competent. This is not data labeling or prompt engineering; it requires deep software engineering judgment.

Responsibilities

• Build believable simulated developer environments that mirror real company codebases and workflows
• Design intermediate-state tasks with clear success criteria that AI agents must solve autonomously
• Write robust test suites that accept all valid solution approaches while rejecting incorrect implementations
• Iterate on tasks and evaluation criteria based on quality assurance feedback and agent performance analysis
• Identify edge cases and failure modes that reveal meaningful differences between AI model capabilities

What We're Looking For

• 5+ years of professional software development experience
• Strong proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka, and Redis
• Proven experience writing functional and integration tests
• Master's degree in Computer Science, Software Engineering, Data Science, AI/ML, NLP, Information Systems, or related field (Bachelor's degree acceptable with 5+ years experience)
• Minimum 3 years in QA automation, testing, or cybersecurity roles
• English proficiency at B2 level or higher
• Deep understanding of where frontier AI models struggle with complex coding tasks
• Ability to distinguish between surface-level correctness and genuinely robust solutions

How We Work With You

Candidates apply through Get A Job.ai, where our talent team will conduct an initial screen. Qualified candidates complete qualification assessments before being submitted to our client. Once accepted, you join project-based work with compensation per completed task. Please do not contact the client directly—all communication flows through our recruitment process.

This is remote freelance work based in Poland, structured to fit around other professional or academic commitments. Payment is per accepted task, with rates determined by your qualification tier and task completion efficiency.

Pay

Compensation is task-based rather than hourly. Rates scale with qualification tier and completion speed, with effective earnings potential reaching competitive freelance rates for senior engineers.

Get A Job.ai is an equal opportunity recruiter. We welcome applications from candidates of all backgrounds and do not discriminate based on race, color, religion, sex, national origin, age, disability, or any other protected characteristic.