QA AI Automation Engineer - LLM and Model Evaluation
Tektora · Abu Dhabi
Job description
About the role
Tektora is supporting an established technology client in Abu Dhabi seeking a QA AI Automation Engineer to own the testing strategy for the company's internal AI and LLM models. This is not a conventional QA role and not about using AI to test software: the thing being tested is the AI itself. The position is permanent, full-time, on-site in Abu Dhabi, five days a week, with relocation support, visa sponsorship and family status available.
Key responsibilities
- Build and maintain Python automation test suites and frameworks from scratch that continuously validate the company's internal AI models
- Test LLM outputs for quality, accuracy, consistency, edge cases and failure modes
- Build and manage test datasets combining real production data with synthetic data, including generated scenarios, edge cases and adversarial inputs
- Write automated test scripts that run on every model and prompt update
- Run regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
- Integrate evaluation into CI/CD with quality gates that block deployment when thresholds drop
- Track and report model performance over time through dashboards and alerting
- Work with the AI, ML and product teams to define what good output looks like per use case and encode it into automated checks
Required profile
- Bachelor's or Master's degree
- Strong Python, including scripting and testing frameworks such as pytest, and API work
- Experience testing AI or LLM outputs or comparable ML system validation, using frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring
- Practical data handling: cleaning real data, generating and synthesising test data, and building golden datasets
- Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
- Basic SQL and NoSQL alongside working data analysis
- Desirable: vector search and embeddings, model fine-tuning concepts, prompt engineering, observability tooling, MCP, GPU/CPU optimisation, ISTQB CT-AI certification
What we offer
- Competitive tax-free package depending on experience
- Full relocation support, visa sponsorship and family status
- Permanent, on-site position in Abu Dhabi
Candidates interested should send their CV through Tektora, who will share the client name and programme details after screening for nationality and location eligibility.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United Arab Emirates.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 2 hours ago
Expires 5 days from now
2 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Tektora
Abu Dhabi
Related job offers
-
Lead Machine Learning Engineer – AI Centre of Excellence
Discovered Abu Dhabi -
Lead Data Scientist – AI Centre of Excellence
Discovered Abu Dhabi -
Vice President – Applications Back‑Office Entreprise
Discovered Abu Dhabi -
Lead Migration de Données d'Entreprise (SAP) – Projet de Transformation Système
Discovered Dubai -
CRM Transformation Manager – Consultant Senior (Middle East)
Discovered Rabat