Principal AI Ops Engineer

About the role

A leading organisation in Abu Dhabi that builds and operates advanced AI systems at scale is looking for a Principal AI Ops Engineer. This is a principal-level individual contributor role combining deep AI infrastructure expertise with strong software and reliability engineering fundamentals. You will own how AI and LLM systems operate in production, covering inference, model serving, deployment, observability, reliability and performance, and set the operational standards that let engineering teams run production AI systems safely, reliably and efficiently.

Key responsibilities

• Design, operate and optimise GPU-based inference and model-serving infrastructure for production AI and LLM workloads.
• Optimise model serving across latency, throughput, batching, quantisation, autoscaling and infrastructure cost.
• Build automated release pipelines for models, prompts and agent configurations, including canary deployments, regression gates and rollback strategies.
• Establish AI-specific observability across model, retrieval and orchestration layers, including tracing, latency, cost and quality monitoring.
• Define and maintain SLOs across availability, latency and AI system quality, with automated alerting and incident response.
• Own capacity planning and cost optimisation across GPU and AI infrastructure.
• Build secure, scalable Kubernetes environments and Infrastructure-as-Code patterns, and develop reusable deployment patterns, tooling and operational standards.
• Lead complex production incidents, load testing and root-cause analysis, and provide technical leadership on engineering standards.

Required profile

• Proven experience at Staff, Principal or equivalent senior IC level, with a track record of running production ML or LLM systems at scale.
• Strong hands-on experience with GPU-based inference and model serving, including vLLM, TGI, TensorRT-LLM or similar.
• Strong reliability engineering fundamentals across SLOs, incident response, capacity planning and post-mortems.
• Experience integrating AI evaluation and regression testing into CI/CD and production releases.
• Experience with cloud infrastructure, ideally Azure, and production environments with strong security, data residency or compliance requirements.
• Experience with GPU/AI infrastructure cost optimisation and FinOps is advantageous.

Required skills

• Strong Python engineering skills with production-grade automation and infrastructure tooling.
• Deep experience with Kubernetes, Docker and Infrastructure as Code, ideally Terraform.
• Observability tools such as Langfuse, LangSmith, Arize Phoenix, Grafana or Prometheus.
• Understanding of batching, quantisation, latency, throughput and autoscaling trade-offs in production LLM serving.