Senior Data Engineer – AI & Data Lakehouse Architecture

About the role

AI71 is seeking a Senior Data Engineer to design and deploy defence‑grade Data Lakehouse platforms that serve as the single source of truth for AI agents supporting EDGE Group’s Intelligent Supply Chain and LeverEDGE generative AI tools.

Key responsibilities

• Architect and implement ingestion pipelines for high‑volume transactional data from SAP S/4HANA, Ariba and PLM systems, delivering near‑real‑time data for forecasting models.
• Build connectors for external market intelligence feeds (e.g., S&P Global, Orbis, EcoVadis) to enrich procurement data with macroeconomic and geopolitical signals.
• Design a unified procurement data model and taxonomy, harmonising fragmented datasets across multiple entities.
• Engineer pipelines to ingest and transform unstructured technical data (PDF tenders, CAD metadata, CONOPS) into vector‑ready formats for Retrieval‑Augmented Generation applications.
• Manage and optimise vector databases such as Weaviate for fast retrieval of embeddings used by AI drafting assistants.
• Establish data lineage and traceability protocols that link requirements to physical components, supporting a digital thread for Model‑Based Systems Engineering.
• Implement defence‑grade security controls (RBAC, audit logging, data redaction) and automated data quality frameworks.
• Optimise pipelines for on‑premise GPU clusters and air‑gapped environments.

Required profile

• Proven experience delivering large‑scale data pipelines for complex ERP environments (SAP S/4HANA, Ariba, PLM).
• Strong background in on‑premise, high‑security data platforms and GPU‑accelerated processing.
• Deep understanding of data governance, security standards and lineage in defence or regulated settings.
• Ability to work in an agile “Sprint Zero” delivery model and collaborate with cross‑functional teams.

Required skills

• Python
• SQL
• Apache Spark
• Kafka
• Airflow
• SAP S/4HANA
• Ariba
• PLM systems
• Weaviate (vector database)
• GPU cluster optimisation