Senior AI Data Engineer - Azure Databricks & Agentic Lakehouse

About the role

Efforts Solutions IT is looking for a Senior Data Engineer (Azure Databricks) to lead the architecture, development and implementation of an enterprise-grade AI-powered Agentic Lakehouse. The role is hands-on technical leadership, working with the AI & Data Manager, internal engineering teams and external delivery partners to turn complex enterprise data into secure, governed and reusable data products. It starts remotely for 2 months, then may require relocation to Abu Dhabi for an onsite or hybrid engagement.

Key responsibilities

• Lead the lakehouse architecture: ingestion, Bronze / Silver / Gold layering, Delta Lake standards and data contracts.
• Build ingestion pipelines from ERP databases, business applications, SharePoint and APIs using Databricks and Azure Data Factory.
• Set up Unity Catalog for governance: catalogs, lineage, access control and data classification.
• Design AI agents that work on the data: schema inference, data-quality rule generation, anomaly detection and entity resolution.
• Expose governed data to AI agents through SQL endpoints, vector indexes and MCP-style data services.
• Embed data-quality checks with quarantine and human review for low-confidence fixes.
• Set engineering standards and review the work of internal engineers and external partners.
• Apply cost controls: cluster policies, compute choices and tagging.

Required profile

• 7+ years in data engineering, including leading lakehouse or data-platform builds.
• Proven track record defining target lakehouse architectures and data engineering standards.
• Experience with data cleansing or migration for ERP systems.
• Practical use of LLMs in data workflows (profiling, DQ rule generation, metadata, text-to-SQL).
• Willingness to relocate to Abu Dhabi after the initial remote period if required.

Required skills

• Azure Databricks: Delta Lake, Unity Catalog, DLT / Lakeflow, Workflows
• Strong PySpark and SQL, data modelling across Medallion layers
• Production ingestion and transformation pipelines with automated data-quality checks
• Azure services: ADLS Gen2, Data Factory, Key Vault, private networking
• Git and CI/CD for data pipelines
• Preferred: knowledge graphs or semantic layers, Databricks Data Engineer Professional certification