Senior/Staff AI Engineer, Quality & Evals (m/f/x)

About This Opportunity

We are representing a well-funded Berlin-based technology company building AI-powered solutions for professional services. They are seeking a Senior or Staff-level AI Engineer to own the evaluation, quality, and observability infrastructure for production LLM agents operating in complex, high-accuracy workflows.

This role sits at the intersection of backend engineering, data infrastructure, and AI systems quality. You will build the evaluation foundation that powers multimodal retrieval agents and drives continuous improvement across critical quality metrics.

This is not a traditional analytics or dashboarding role. You will write production code, design data architecture, work within backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents.

Responsibilities

• Build online and offline evaluation systems for LLM agents, including pipelines using golden datasets, ground-truth data, human review workflows, and experiment results
• Create automated quality gates to test changes to prompts, context, models, or agent logic before production deployment
• Analyze large volumes of agent traces and executions in columnar and analytical databases (BigQuery, ClickHouse) to identify failure modes, quality regressions, latency issues, and cost optimization opportunities
• Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior
• Manage observability tools for tracing, monitoring, debugging, and experiment management
• Collaborate with backend engineers to improve the speed and reliability of retrieval and reasoning agents

What We're Looking For

Required qualifications:

• Strong Python and/or backend engineering experience at a senior level
• Solid understanding of LLM and agent system evaluation approaches—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics
• Experience deploying and operating cloud systems, ideally on GCP
• Hands-on experience building end-to-end retrieval or ML pipeline evaluation systems
• Experience with LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases
• Comfort working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data
• Understanding of system design, reliability, observability, monitoring, logging, and debugging
• Senior-level engineering judgment with ability to make architectural decisions and communicate trade-offs
• Comfort with ambiguity and ability to reason from first principles

Valued additional experience:

• Designing data pipelines, ETL/ELT workflows, or event-processing systems
• Building infrastructure around LLM-based products or agentic systems
• Optimizing LLM usage, context windows, reasoning tokens, or model selection
• Working with production traces from complex distributed systems
• Building internal platforms for engineers or domain experts
• Using workflow orchestration systems such as Temporal
• Experience in high-accuracy domains like audit, finance, or compliance
• Early-stage startup or fast-moving engineering environment experience

How We Work With You

When you apply through Get A Job.ai, our talent team will review your profile and conduct an initial screening. If there's a strong match, we'll submit your candidacy to our client for consideration. The client's interview process includes technical discussions, cultural fit conversations, and an on-site day in Berlin to meet the team and work on real problems together.

Please apply exclusively through Get A Job.ai and do not contact the employer directly. Our recruiters will guide you through each stage and provide feedback throughout the process.

Location & Work Arrangement

This position is based in Berlin, Germany. Our client values in-person collaboration and operates from a central Berlin office.

Pay

Compensation details will be discussed during the screening process and vary based on experience level and qualifications.

Equal Employment Opportunity: Get A Job.ai is committed to inclusive hiring practices. We encourage applications from candidates of all backgrounds and work with clients who share our commitment to diversity and equal opportunity.