Product Manager – Data & ML Evaluation (Enterprise)

About the role

A client of Black Pearl Consult is looking for a highly analytical and technically strong Product Manager to lead the strategy and execution of product data, experimentation and machine learning evaluation. The role acts as product owner for data and evaluation frameworks, making sure AI-driven products are measurable, reliable and continuously improving, working with Product, Engineering, Data Science and business stakeholders.

Key responsibilities

• Own and manage the end-to-end product data ecosystem: telemetry, tracking plans, event schemas, data quality and warehousing
• Define and implement a product metrics framework (north star metrics, leading indicators, input metrics) and provide dashboards and insights to stakeholders
• Design and run machine learning evaluation frameworks, including test set curation and KPIs for quality, safety, bias, latency and cost
• Run and optimise offline and online evaluations, build reproducible evaluation pipelines integrated with CI/CD, and lead A/B testing and experimentation
• Drive data-informed pricing strategies based on usage patterns, feature value and cost structures
• Establish data governance standards (event naming, privacy, PII handling, data retention) and collaborate with Security, Legal and business teams on compliance

Required profile

• 5-8+ years of experience in Product Management, with a strong background in data, analytics or engineering
• Proven experience owning product analytics, instrumentation and experimentation frameworks
• Hands-on experience with machine learning products, including model evaluation and monitoring
• Excellent stakeholder management and communication skills, able to work in fast-paced, cross-functional environments

Required skills

• Strong SQL and data analysis, able to validate and interpret experiments independently
• Understanding of A/B testing and statistical concepts (power, MDE, guardrail metrics)
• Preferred: product analytics tools (e.g. Amplitude), experiment tracking (e.g. MLflow), data modelling tools (e.g. dbt), feature stores, observability platforms, LLM/RAG evaluation frameworks, and comfort using AI tools to prototype analyses