AI Engineer – Agent Learning & Evaluation

About the role

We are building an enterprise AI platform where AI agents operate business applications through their user interface, just like a human would. As an AI Engineer you will own the learning mechanisms that keep agents up‑to‑date and the evaluation framework that proves their reliability before deployment.

Key responsibilities

• Design and implement methods for agents to learn applications from human demonstrations and self‑exploration, converting knowledge into reusable playbooks.
• Represent agent knowledge as structured data such as knowledge graphs of screens, actions, and data.
• Build simulated and recorded environments for safe agent practice and testing.
• Develop self‑correction capabilities to detect application changes and update agent knowledge without breaking production tasks.
• Create an evaluation framework with task suites, success metrics, LLM‑as‑judge calibration, regression testing, and online reliability monitoring.
• Improve agent performance via prompt/program optimisation, fine‑tuning, or reinforcement learning where beneficial.
• Collaborate closely with the AI Engineer – Agent Harness and mentor senior engineers on evaluation best practices.

Required profile

• 10+ years of software engineering experience, including at least 2 years building LLM‑powered applications.
• Proven experience designing and running rigorous evaluations for LLM or agent systems.
• Strong Python programming skills and comfort with statistics and experimental design.
• Demonstrated technical leadership: architecture decisions, mentoring, and raising engineering standards.

Required skills

• Python
• Transformer architecture
• Fine‑tuning and reinforcement learning for LLMs
• Graph databases (e.g., Neo4j, Memgraph)
• Agent memory and learning from demonstration
• Recorded browser session handling (HAR/WARC, rrweb)
• LLM‑as‑judge evaluation tools (e.g., Langfuse, Arize Phoenix)