AI Quality & Reliability Engineer (QA/SRE)

About the role

An AI Quality & Reliability Engineer is sought to own the production readiness of cloud-based AI solutions. This hybrid role combines automated software testing, AI model evaluation and Site Reliability Engineering (SRE), building automated frameworks to validate AI outputs while keeping Azure/AWS infrastructure resilient, performant and aligned with banking standards.

Key responsibilities

• Design and run automated testing frameworks for AI services such as Azure OpenAI and AWS Bedrock, covering model hallucinations, accuracy, content safety and latency.
• Build CI/CD-integrated test suites in Python to validate application logic and infrastructure state.
• Implement chaos engineering and load testing for banking-scale web and mobile backends, and develop automated recovery scripts for high availability.
• Define and monitor SLIs and SLOs, and configure monitoring and alerting with Azure Monitor or AWS CloudWatch.
• Automate security scans and compliance checks, and ensure AI data handling meets banking data residency and privacy requirements.
• Identify inefficient cloud resource usage and expensive failing tests during testing.

Required profile

• 7+ years of relevant experience.
• Strong hands-on experience with Azure or AWS, especially networking, scaling and serverless reliability.
• Understanding of prompt engineering and AI output evaluation, including RAG evaluation, ROUGE/BLEU and custom LLM benchmarks.

Required skills

• Python and automation frameworks such as PyTest, Selenium and Robot Framework.
• Grafana, Prometheus, Azure Monitor, AWS CloudWatch or similar monitoring tools.
• Bash, GitHub Actions, Terraform, K6/JMeter.
• AI evaluation tools such as DeepEval, Ragas or LangSmith.