Research Scientist / Engineer – Training Infrastructure

About the role

You will build distributed systems that train Luma's large‑scale multimodal models across thousands of GPUs, enabling researchers to focus on innovation on top of reliable, efficient, and scalable infrastructure.

Key responsibilities

• Design, implement, and optimize efficient distributed training systems for models across thousands of GPUs.
• Research and implement advanced parallelization techniques such as FSDP, Tensor Parallel, Pipeline Parallel, and Expert Parallel.
• Build monitoring, visualization, and debugging tools for large‑scale training runs.
• Optimize training stability, convergence, and resource utilization across massive clusters.

Required profile

• Extensive experience with distributed PyTorch training and parallelism in foundation‑model training.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with communication libraries like NCCL and MPI.
• Strong Linux systems administration and scripting skills.

Required skills

• PyTorch, CUDA, FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel
• NCCL, MPI
• Linux administration, Bash/Python scripting
• Containerization (Docker), orchestration (Kubernetes), cloud infrastructure