AI Architect - Data Pipelines and Spark

About the role

This role is for an engineer who takes ownership of data pipelines from design through production support, writing clean, maintainable and well-tested code and building reusable, production-ready solutions. The work centres on batch and real-time data pipelines, data products and streaming solutions.

Key responsibilities

• Design, build and maintain batch and real-time data pipelines.
• Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• Build streaming solutions with Spark Structured Streaming and Lakeflow declarative pipelines.
• Optimise pipelines for scalability, reliability and performance.
• Develop automated tests and take part in code reviews.
• Contribute to CI/CD and engineering best practices.
• Monitor, troubleshoot and support production workloads.
• Use AI-assisted development tools to improve productivity while keeping high engineering standards.

Required profile

• Strong analytical and problem-solving skills.
• Ability to collaborate effectively with teammates and stakeholders.
• Nice to have: experience in Agile or SAFe environments.

Required skills

• Databricks Lakeflow Declarative Pipelines and DXQ, DB Genie, MCP Server
• Apache Spark (PySpark) and Spark Structured Streaming
• Auto Loader, Unity Catalog, Python, SQL, Delta Lake
• Azure Event Hubs or equivalent event streaming platforms
• Git, unit testing and CI/CD
• Nice to have: performance tuning and Spark optimisation, Azure DevOps, GitHub Copilot or similar AI coding assistants