Vision-Language Model (VLM) Engineer

About the role

The Technology Innovation Institute (TII), a publicly funded research institute in Abu Dhabi, UAE, is hiring a VLM Engineer within its Artificial Intelligence and Digital Research Centre. The Extreme-Scale Language Model team develops deep learning technologies covering natural language processing, perception and vision, having built the Falcon models and now continuing research on large language models.

Key responsibilities

• Conduct ablation studies on vision models to assess the impact of components and configurations, collaborating with researchers to analyse and report on model architectures and settings.
• Carry out data ablation studies with the team to identify optimal data types and structures for training vision-language models, and analyse how different data inputs affect performance, especially vision-language alignment.
• Develop and run robust evaluation protocols, assessing model performance across benchmarks and real-world scenarios.
• Take part in model training, with an emphasis on integrating LLMs with vision models such as CLIP.

Required profile

• PhD in deep learning.
• Proven research and development experience in vision-language models.
• A publication record in top-tier conferences is highly desirable.

Required skills

• Machine learning expertise, especially vision-language models and LLMs.
• Understanding of architectures such as CLIP and their use in vision-language tasks.
• Distributed training and multi-GPU optimisation.
• Deep learning frameworks such as PyTorch.
• Analytical skills for ablation studies and model evaluation.
• Familiarity with curating and processing datasets for vision and language tasks.