Senior QA Automation Engineer

About the role

We are seeking a Senior QA Automation Engineer to lead validation and verification of AI systems for mission‑critical defense applications. You will design automated testing frameworks for large language models and predictive engines, ensuring they meet strict reliability and compliance standards.

Key responsibilities

• Architect and implement automated testing frameworks for non‑deterministic AI outputs, including hallucination detection, consistency checks, and factual accuracy against gold‑standard datasets.
• Design evaluation metrics (RAGAS, faithfulness, answer relevance) for Retrieval‑Augmented Generation pipelines and create prompt regression suites to monitor drift.
• Develop integration tests to validate data consistency between AI agents and enterprise systems such as SAP S/4HANA and Ariba.
• Build performance benchmarking tests for latency and throughput of forecasting and risk‑scoring models.
• Automate API validation, ensuring RBAC and PII redaction logic before data ingestion.
• Prepare Test Readiness packages and manage defect lifecycle, linking issues to specific model versions or datasets.

Required profile

• 5+ years of QA automation experience, with at least 2 years testing data‑driven or AI applications.
• Experience in defense, aerospace, or other highly regulated sectors is a strong plus.
• Understanding of IV&V processes and ability to define pass/fail criteria for probabilistic systems.

Required skills

• Python (Pytest, Selenium, Playwright, Requests)
• Performance testing tools: Locust, JMeter, K6
• LLM evaluation frameworks: DeepEval, TruLens or custom Python evaluators
• SQL and data validation tools such as Great Expectations
• GitLab CI/CD pipelines with quality‑gate enforcement
• Requirements management tools: Jira, Linear, Jama, Polarion
• Test reporting tools: TestRail, Allure
• Version control: Git, GitLab