Senior Data Engineer – AI & Data Lakehouse

About the role

AI71 is looking for a Senior Data Engineer to design and implement the core data infrastructure that will power EDGE Group’s AI transformation. You will build defense‑grade Data Lakehouses that act as the single source of truth for AI agents, supporting intelligent supply‑chain forecasting and generative AI tools.

Key responsibilities

• Architect and deploy ingestion pipelines for high‑volume transactional data from SAP S/4HANA, Ariba and PLM systems, ensuring near real‑time availability for forecasting models.
• Build connectors for external market intelligence feeds such as S&P Global, Orbis and EcoVadis to enrich procurement data.
• Design a unified procurement data model and taxonomy across multiple entities.
• Engineer pipelines to ingest and process unstructured technical data (PDFs, CAD metadata, CONOPS) into vector‑ready formats for Retrieval‑Augmented Generation applications.
• Manage and optimise vector databases (e.g., Weaviate) for fast retrieval of embeddings used by AI drafting assistants.
• Implement data lineage and traceability protocols that link requirements to physical components, supporting a digital thread for Model‑Based Systems Engineering.
• Enforce defense‑grade security, RBAC, audit logging and data redaction to meet export‑control and on‑premise requirements.
• Deploy automated data quality frameworks to validate BOM completeness and cost accuracy before data reaches AI models.
• Optimise pipelines for on‑premise GPU clusters and air‑gapped environments.

Required profile

• Proven experience designing and operating large‑scale data lakehouse architectures, preferably in supply‑chain or defense contexts.
• Strong background integrating complex ERP systems and external data feeds.
• Hands‑on experience with security and governance in highly regulated environments.
• Ability to work in a sprint‑zero, agile setting and collaborate across multidisciplinary teams.

Required skills

• Python
• SQL
• Apache Spark
• Kafka
• Airflow
• Weaviate (vector database)