Principal AI Engineer – Context – Agents and Context

About This Opportunity

We are representing a confidential search platform company in their search for a Principal AI Engineer to lead the improvement loop for their AI agent infrastructure. This is a remote position working with a distributed engineering team building the knowledge layer that powers enterprise AI agents.

In this role, you will own how AI agents and automations behave in production—designing the evaluation frameworks, telemetry systems, and safe deployment strategies that allow the team to iterate on agentic behavior without breaking customer integrations. You'll work at the intersection of engineering, data science, and product, writing production TypeScript code while setting the technical bar for eval-driven development across the team.

Responsibilities

• Own the production improvement loop for the AI context engine: understand how extraction automations, retrieval tools, and memory systems behave based on offline evaluations, customer feedback, and telemetry data
• Define safe iteration practices for agents and skills: establish versioning and rollout processes for prompts, skills, and automations, including regression coverage, staged evaluation, and guardrails
• Design telemetry architecture to enable data-informed decisions: determine what to capture from agent traces, tool calls, and knowledge retrieval, and how it feeds evaluation and feedback loops
• Partner with data science teams on evaluation strategy: develop golden datasets and evaluators to gate on quality, latency, and cost
• Raise the engineering bar: review designs and pull requests, mentor engineers in eval-driven development, and author technical proposals that shape the roadmap
• Build production features in TypeScript for APIs, stateful workflows, and data pipelines that serve agent platforms

What We're Looking For

• 10+ years of software engineering experience, with recent years shipping and operating AI-driven products on production traffic
• Track record of eval-driven product improvement: diagnosing agent or LLM behavior from traces, designing evaluations, shipping fixes, and measuring outcomes
• Direct experience building agents with state and memory, and iterating on prompts, skills, and tool behavior safely in production
• Familiarity with MCP (Model Context Protocol), including exposing public MCP servers and tools
• Experience designing telemetry for AI systems and using it to make engineering and product decisions
• Experience running product experiments end to end: instrumentation, unattended execution, and interpreting results
• Experience building products with public APIs and evolving data models while maintaining backward compatibility
• Strong backend engineering skills in Python or TypeScript: APIs, stateful workflows, data pipelines, and production services
• Comfort collaborating with data scientists, engineers, and product managers, translating between measurement and shipping
• Pragmatic, low-ego communication style suited to distributed, async-first teams

Bonus qualifications:

• Experience with agent frameworks such as LangGraph, CrewAI, Claude Agent SDK, or similar
• Agentic retrieval experience, including knowledge representation
• Practical experience with enterprise search platforms

How We Work With You

Candidates apply through Get A Job.ai. Our talent team will conduct an initial screening to understand your background and match it to our client's requirements. If there's a strong fit, we will submit your profile to the client for consideration. Please do not attempt to contact the client directly—all communication should go through our recruiting team to ensure a smooth process.

Pay

Compensation details will be discussed during the screening process and are competitive for principal-level engineering roles at distributed technology companies.

Equal Opportunity: Get A Job.ai is committed to inclusive recruiting practices. We welcome candidates of all backgrounds and provide equal consideration regardless of race, ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, marital status, veteran status, or disability status.