Research Engineer (Evals)

About the Opportunity

We are representing a confidential AI safety organization seeking a Research Engineer to own and scale their internal evaluation infrastructure. This role sits at the intersection of research and production, building benchmarks that measure how AI systems succeed and fail in real-world conditions.

You'll join a fundamental research team studying agent reliability, guardrail effectiveness, and model behavior at scale. Your benchmarks will directly inform product decisions and, in some cases, become public research contributions.

Responsibilities

• Design, build, and maintain comprehensive benchmark suites covering single-turn, multi-turn, and agentic safety scenarios
• Develop evaluation frameworks that produce measurable, defensible capability distinctions between models
• Collaborate with product teams to create evals covering core model functionality and new feature releases
• Generate synthetic data for post-training evaluation of textual and multimodal systems
• Adapt existing benchmarks to new verticals and evolving product requirements
• Conduct research projects quantifying realistic LLM and agent failure modes in production environments
• Build and optimize efficient LLM inference pipelines with proper orchestration, retry logic, and rate-limit handling

What We're Looking For

Required:

• Demonstrated experience building LLM benchmarks from scratch that successfully distinguished specific model capabilities with measurable results
• Hands-on experience creating synthetic training data for language or multimodal models
• Ability to reproduce published benchmark results and critically assess methodology weaknesses
• Strong Python engineering skills with production code experience—you write maintainable code that others can extend
• Proficiency building efficient LLM inference systems including parallel orchestration and robust error handling
• Daily fluency with frontier AI models and coding agents

Bonus qualifications:

• Experience with automated red-teaming approaches
• Track record working across multiple agentic frameworks and reproducing public benchmark results
• Deep familiarity with reward-model, monitoring, and safety evaluation benchmarks
• Published research in evaluations or safety-evaluation domains

How We Work with You

Our talent team at Get A Job.ai partners with companies building critical AI infrastructure. When you apply through our platform, a specialist recruiter will conduct an initial screen to understand your background and ensure alignment with the role. We then coordinate your introduction to our client and guide you through their process, which includes a take-home evaluation task, a technical interview with the research leadership, and a final conversation with executive leadership.

Candidates should not contact the client directly. All communication flows through Get A Job.ai to ensure a structured, professional experience.

Location & Work Arrangement

This position is based in Paris with a hybrid work model. Relocation support is available for qualified candidates.

Pay

Compensation details will be discussed during the screening process and are competitive with European AI research engineering markets. The package includes meaningful equity participation.

Get A Job.ai is an equal opportunity recruiter. We work with clients and candidates without regard to race, color, religion, sex, national origin, age, disability, or any other protected characteristic.