Data Engineer - Azure & Databricks

About the role

A Data Engineer is sought to build, enhance and support scalable data solutions on Azure. The role covers data pipelines, integration of SAP and non-SAP sources, transformations in Databricks and delivery of trusted datasets for reporting, analytics and business intelligence.

Key responsibilities

• Develop and maintain data ingestion pipelines in Azure Data Factory and integrate data from SAP, databases, APIs, flat files and cloud apps, including incremental and CDC-based loading
• Build PySpark and SQL transformations in Azure Databricks, maintain Bronze, Silver and Gold layers, and cleanse, validate and enrich data
• Support enterprise data models and Common Data Models, and deliver curated datasets for reporting and analytics
• Provide trusted datasets for Power BI and work with Business Analysts, BI Developers and Data Architects; support report and dashboard performance
• Implement data quality checks, follow governance and security standards, document data flows and business rules
• Monitor jobs and pipelines, investigate production incidents, perform root cause analysis and take part in deployments

Required profile

• Bachelor's degree in Computer Science, IT, Engineering or a related field
• Overall 5-8 years of experience, 3-6 years in data engineering, 2+ years with Azure Databricks and Azure Data Factory
• Experience building enterprise-scale cloud data pipelines
• Preferred: SAP ECC / S/4HANA integration, REST API integrations, event-driven architectures, Unity Catalog, Microsoft Fabric, retail or automotive domain experience

Required skills

• Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Delta Lake
• Python, PySpark, SQL
• ETL / ELT, data warehousing, medallion architecture, incremental loading, dimensional modeling
• Power BI, Git, Azure DevOps, CI/CD concepts