Senior QA Automation Engineer – AI Validation

About the role

AI71 is looking for a Senior QA Automation Engineer to lead validation and verification of AI systems for EDGE Group’s defense‑sector transformation. You will define quality standards for non‑deterministic AI models, bridge Agile development with formal Systems Engineering, and ensure AI agents meet rigorous Test Readiness Reviews and Functional Configuration Audits.

Key responsibilities

• Design and implement automated frameworks to test Large Language Model outputs for hallucination, consistency, and factual accuracy against gold‑standard datasets.
• Develop RAG evaluation metrics (e.g., RAGAS, faithfulness, answer relevance) to verify Retrieval‑Augmented Generation pipelines.
• Create regression suites that monitor prompt drift and model version changes.
• Build integration tests for ERP systems such as SAP S/4HANA and Ariba to guarantee data integrity.
• Design performance benchmarks for forecasting and risk‑scoring models, ensuring latency and throughput meet real‑time dashboard requirements.
• Automate API gateway testing, validating RBAC controls and PII redaction before data reaches AI models.
• Map test cases to system requirements and user needs, preparing test‑readiness packages for Stage Gate reviews.
• Manage defect lifecycle, linking AI logic defects to specific model versions or data sets.

Required profile

• Senior‑level professional with extensive experience in test automation for AI/ML systems.
• Strong understanding of V‑Model, Stage Gate processes, and formal verification & validation.
• Proven ability to work across Agile development and Systems Engineering teams.

Required skills

• Python programming (Pytest, custom test harnesses).
• Experience with SAP S/4HANA and Ariba integration testing.
• API security testing, including RBAC and PII redaction.
• Knowledge of RAG evaluation metrics such as RAGAS.
• Performance and latency benchmarking for AI models.
• Automation of non‑deterministic AI testing and prompt regression.