Senior Data Engineer - Databricks

About the role

PACT Employment is recruiting a Senior Data Engineer in Abu Dhabi on a contractor basis. The role focuses on migrating and optimising data pipelines from Foundry to Databricks, including performance tuning and integration with a Palantir reconciliation engine.

Key responsibilities

• Implement Databricks pipelines using Delta Live Tables (declarative pipelines).
• Rewrite proprietary Foundry transforms into optimised PySpark on Databricks.
• Move from Foundry's build system to modern CI/CD with IaC-managed Databricks workspaces.
• Tune Databricks Photon performance to match or exceed Foundry interactivity.
• Contribute to the reconciliation engine ensuring Palantir and Databricks outputs are equivalent.
• Handle pipeline rebuilds, validation, internal testing and UAT preparation.

Required profile

• Hands-on Databricks experience (Delta Live Tables, Unity Catalog, Workflows) in production, not just in a lab.
• Strong English, written and spoken, for direct interaction with Databricks Professional Services.

Required skills

• PySpark and advanced SQL.
• Data modelling and schema design (medallion / lakehouse).
• Modern CI/CD for data pipelines (IaC, Git-based deployment).