Staff AI Product Analyst

About This Opportunity

We are representing a confidential technology client in their search for a Staff AI Product Analyst to build and lead their AI-native product analytics capability. This is a senior individual contributor role with significant cross-functional influence, designing evaluation frameworks, defining measurement standards, and establishing the data ecosystem that enables both human teams and AI agents to work from a foundation of trust and accuracy.

Our client is a mature platform company serving millions of users. They are investing heavily in agentic AI systems and need a systems-thinking data expert who can bridge deep AI capabilities with practical business outcomes.

Responsibilities

• AI Product Strategy & Metrics Design: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes. Collaborate on test design and causal framing for new AI product launches.
• Data Layer Systems Thinking: Map how agents interact with metrics, raw tables, documentation, and APIs to ensure analytics and data engineering requirements are explicit, structured, and testable.
• Evaluation Framework Architecture: Design and maintain formal evaluation sets, golden datasets, rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness.
• Taxonomy & Failure Mode Analysis: Systematically categorize errors, tool misuse, context gaps, and data edge cases, feeding insights back into prompts, retrieval systems, tools, and data modeling pipelines.
• Performance Monitoring: Track live usage of agentic AI analytics workflows—monitoring success rates, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog.
• Cross-Functional Leadership: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Partner with business leaders and engineering teams to translate AI capabilities into measurable outcomes.
• Measurement Artifacts: Deliver recurring dashboards, reviews, or written readouts that leadership uses to prioritize roadmaps while reducing high-severity incidents involving incorrect metrics.

What We're Looking For

Required Qualifications:

• Bachelor's degree in a quantitative field (Computer Science, Statistics, Mathematics, or related)
• 10+ years of experience in product analytics, data science, or technical product management within complex, high-scale environments
• 1+ years of direct, hands-on experience working with AI/LLM systems in applied contexts, with specific focus on evaluation, trust, and responsible deployment
• Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices
• Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, and production monitoring
• Exceptional communication skills with ability to translate technical AI insights into clear narratives for non-technical stakeholders

Preferred Qualifications:

• Experience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systems
• Demonstrated ability to operate in ambiguity, run structured experiments, and default to fast action and iteration
• Systems-oriented mindset with healthy skepticism of AI hype, focusing on organizational outcomes and technical investments that yield true business value

How We Work With You

Our talent team at Get A Job.ai partners with leading technology companies to identify exceptional candidates for confidential search engagements. When you apply through our platform, a specialized recruiter will conduct an initial screening to understand your background and assess fit. Qualified candidates will then be submitted directly to our client for consideration. Please do not attempt to contact the client directly, as all communication is managed through Get A Job.ai to ensure a professional and confidential process.

Pay

Pay via Get A Job.ai: $165,000 - $225,000 USD, commensurate with experience and performance. This position is also eligible for a competitive year-end performance bonus and equity package.

Location: USA (Remote)

Get A Job.ai is an equal opportunity recruiter. We welcome and value individuals who bring diverse life experiences, educational backgrounds, cultures, and work experiences.