QA AI Automation Engineer - LLM and Model Evaluation
Tektora · Abu Dhabi
وصف الوظيفة
About the role
Tektora is supporting an established technology client in Abu Dhabi seeking a QA AI Automation Engineer to own the testing strategy for the company's internal AI and LLM models. This is not a conventional QA role and not about using AI to test software: the thing being tested is the AI itself. The position is permanent, full-time, on-site in Abu Dhabi, five days a week, with relocation support, visa sponsorship and family status available.
Key responsibilities
- Build and maintain Python automation test suites and frameworks from scratch that continuously validate the company's internal AI models
- Test LLM outputs for quality, accuracy, consistency, edge cases and failure modes
- Build and manage test datasets combining real production data with synthetic data, including generated scenarios, edge cases and adversarial inputs
- Write automated test scripts that run on every model and prompt update
- Run regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
- Integrate evaluation into CI/CD with quality gates that block deployment when thresholds drop
- Track and report model performance over time through dashboards and alerting
- Work with the AI, ML and product teams to define what good output looks like per use case and encode it into automated checks
Required profile
- Bachelor's or Master's degree
- Strong Python, including scripting and testing frameworks such as pytest, and API work
- Experience testing AI or LLM outputs or comparable ML system validation, using frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring
- Practical data handling: cleaning real data, generating and synthesising test data, and building golden datasets
- Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
- Basic SQL and NoSQL alongside working data analysis
- Desirable: vector search and embeddings, model fine-tuning concepts, prompt engineering, observability tooling, MCP, GPU/CPU optimisation, ISTQB CT-AI certification
What we offer
- Competitive tax-free package depending on experience
- Full relocation support, visa sponsorship and family status
- Permanent, on-site position in Abu Dhabi
Candidates interested should send their CV through Tektora, who will share the client name and programme details after screening for nationality and location eligibility.
Questions fréquentes
لماذا تبلغ عن هذا العرض؟
اكتشف المزيد
الرواتب والأدلة وعمليات البحث في الإمارات العربية المتحدة.
الرواتب حسب المهنة
قدم طلبك في 30 ثانية
أدخل بريدك الإلكتروني للتقديم. سيتم إنشاء حساب تلقائياً.
بالمتابعة، أنت توافق على شروط الاستخدام.
لديك حساب بالفعل؟ تسجيل الدخول
لديك سؤال حول هذا العرض؟
اطرحه هنا: ستصلك تفاصيل العرض كاملة عبر البريد الإلكتروني، فوراً.
عزز فرصك
حمّل سيرتك الذاتية وسنقترح عليك الوظائف التي تناسب ملفك.
جاري تحليل سيرتك الذاتية...
Tektora
Abu Dhabi
عروض عمل ذات صلة
-
Lead Machine Learning Engineer – AI Centre of Excellence
Discovered Abu Dhabi -
Lead Data Scientist – AI Centre of Excellence
Discovered Abu Dhabi -
Vice President – Applications Back‑Office Entreprise
Discovered Abu Dhabi -
Lead Migration de Données d'Entreprise (SAP) – Projet de Transformation Système
Discovered Dubai -
CRM Transformation Manager – Consultant Senior (Middle East)
Discovered Rabat