Azure Databricks Data Engineer (Cloud Engineer)

About the role

We are looking for an experienced Azure Databricks Data Engineer to design, build and maintain reliable, scalable data pipelines on the Databricks Lakehouse platform. The role will support analytics, operational reporting and the Data Intelligence Layer for our Abu Dhabi office.

Key responsibilities

• Build batch pipelines using Delta Lake with optimized table design, partitioning, Z‑ORDER, OPTIMIZE and VACUUM strategies.
• Implement incremental ingestion with Databricks Auto Loader, schema evolution and checkpointing.
• Develop Structured Streaming pipelines with watermarking, state management and restart safety.
• Design Lakeflow pipelines for governed, declarative data processing and ensure replayable, idempotent pipelines for safe backfills.
• Optimize Spark workloads via Adaptive Query Execution, shuffle tuning, skew mitigation and join optimization.
• Create analytics‑ready datasets consumed by Databricks SQL, dashboards, Genie and downstream applications.
• Develop DBSQL models and semantic data layers aligned with Data Intelligence requirements.
• Package and deploy solutions using Databricks Repos and Asset Bundles.
• Support Synapse‑to‑Databricks migration initiatives and performance benchmarking.

Required profile

• 8+ years of professional experience in data engineering.
• Strong expertise in Azure Databricks, Lakehouse architecture and cloud‑native data processing frameworks.
• Proven ability to design governed, replayable and idempotent pipelines.

Required skills

• Azure Databricks
• Azure Data Factory (ADF)
• Unity Catalog
• Databricks Asset Bundles
• Medallion Architecture
• PySpark
• SQL & Databricks SQL (DBSQL)
• Azure Synapse Analytics
• Real‑Time Streaming & Structured Streaming
• Lakeflow Pipelines
• Delta Lake
• Data Intelligence Layer implementation
• CI/CD & Databricks Repos
• Performance optimization & query tuning

What we offer

• Employee visa for Abu Dhabi
• Medical insurance
• Annual airfare
• 15 days accommodation allowance