Data Engineer – Scalable Autonomous Systems

About the role

This role focuses on designing and building scalable data infrastructure to support advanced autonomous systems. You will transform large‑scale multimodal sensor data into high‑quality, structured datasets ready for downstream processing and machine‑learning workflows.

Key responsibilities

• Design and manage high‑throughput on‑vehicle recording pipelines (multi‑GB/s, MCAP/rosbag2) with accurate timestamp synchronization.
• Oversee on‑platform storage and reliable transfer of data to cloud environments, ensuring integrity checks.
• Architect and maintain scalable data lake solutions across S3, FSx/Lustre, and GCS, including data organization, caching, retention, lineage, and cost optimization.
• Build pipelines that convert raw sensor streams into training‑ready datasets with correct time alignment, calibration metadata, and scenario tagging.
• Implement robust dataset versioning and discovery using tools such as DVC, LakeFS, Deep Lake, and FiftyOne.
• Contribute to efficient on‑disk dataset format design for high‑performance I/O.
• Develop and manage annotation workflows, handling vendor handoff formats, quality control, schema evolution, and iterative dataset improvements.

Required profile

• 5+ years of experience building production‑grade data infrastructure, preferably with large‑scale multimodal or sensor data.
• Strong proficiency in Python and ability to work with C++ for ROS2 tooling.
• Hands‑on experience with cloud storage and distributed file systems (S3, GCS, FSx/Lustre) and performance/cost optimization.
• Experience with dataset versioning and ML data management tools.

Required skills

• Python
• C++
• ROS2
• MCAP / rosbag2
• Amazon S3
• Google Cloud Storage
• FSx / Lustre
• DVC
• LakeFS
• Deep Lake
• FiftyOne
• Distributed Data Parallel (DDP)
• Fully Sharded Data Parallel (FSDP)