📢 New: get today's jobs on our WhatsApp Channel
Jobiglo

No results.

QA AI Automation Engineer - LLM and Model Evaluation

Tektora · Abu Dhabi

New
دائم Onsite Senior 🇬🇧 English
Python pytest RAGAS DeepEval LangSmith GitLab CI GitHub Actions SQL NoSQL FastAPI Docker Linux

Job description

About the role

Tektora is supporting an established technology client in Abu Dhabi seeking a QA AI Automation Engineer to own the testing strategy for the company's internal AI and LLM models. This is not a conventional QA role and not about using AI to test software: the thing being tested is the AI itself. The position is permanent, full-time, on-site in Abu Dhabi, five days a week, with relocation support, visa sponsorship and family status available.

Key responsibilities

  • Build and maintain Python automation test suites and frameworks from scratch that continuously validate the company's internal AI models
  • Test LLM outputs for quality, accuracy, consistency, edge cases and failure modes
  • Build and manage test datasets combining real production data with synthetic data, including generated scenarios, edge cases and adversarial inputs
  • Write automated test scripts that run on every model and prompt update
  • Run regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
  • Integrate evaluation into CI/CD with quality gates that block deployment when thresholds drop
  • Track and report model performance over time through dashboards and alerting
  • Work with the AI, ML and product teams to define what good output looks like per use case and encode it into automated checks

Required profile

  • Bachelor's or Master's degree
  • Strong Python, including scripting and testing frameworks such as pytest, and API work
  • Experience testing AI or LLM outputs or comparable ML system validation, using frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring
  • Practical data handling: cleaning real data, generating and synthesising test data, and building golden datasets
  • Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
  • Basic SQL and NoSQL alongside working data analysis
  • Desirable: vector search and embeddings, model fine-tuning concepts, prompt engineering, observability tooling, MCP, GPU/CPU optimisation, ISTQB CT-AI certification

What we offer

  • Competitive tax-free package depending on experience
  • Full relocation support, visa sponsorship and family status
  • Permanent, on-site position in Abu Dhabi

Candidates interested should send their CV through Tektora, who will share the client name and programme details after screening for nationality and location eligibility.

Questions fréquentes

Le salaire n'est pas communiqué publiquement par le recruteur. Vous pouvez postuler et négocier directement avec Tektora.
Cliquez sur "Postuler maintenant" en haut de la page. Vous pouvez importer votre CV en 1 clic — Jobiglo extrait automatiquement vos informations et postule pour vous.
Le contrat proposé est un دائم basé à Abu Dhabi.

Why are you reporting this job?

Thank you for your report. We will review this job.

Apply in 30 seconds

Enter your email to apply. An account will be created automatically.

Apply now →

By continuing, you accept our terms of use.

Already have an account? Login

A question about this job?

Ask it here: you will get the full job summary by e-mail, right away.

Published 2 hours ago

Expires 5 days from now

2 views · 0 interested

Boost your chances

Upload your CV — we will match you with relevant openings.

Analyzing your CV...

Tektora

Abu Dhabi