Data Engineer (PySpark)

About the role

Black Pearl Consult is looking for an experienced Data Engineer (PySpark) to design, build, optimize and maintain scalable data pipelines for production environments. The role calls for strong hands-on experience in big data processing, pipeline optimization and deployment with modern data engineering tools.

Key responsibilities

• Design, develop and maintain robust, scalable data pipelines using Python and PySpark
• Perform data ingestion, transformation, cleansing and validation across structured and unstructured datasets
• Run exploratory data analysis to find patterns, anomalies and quality issues, and apply imputation, linking and cleansing
• Build feature engineering pipelines for analytics and downstream use cases
• Optimize Spark jobs for performance, scalability and cost efficiency
• Deploy and tune production-grade pipelines, and automate workflows with Apache Airflow and/or Jenkins
• Collaborate with cross-functional teams to integrate data solutions into production systems
• Write and maintain unit tests, and manage source code, CI/CD and deployments with Git, GitHub and GitHub Actions

Required profile

• Previous production experience is mandatory, specifically deploying, tuning and maintaining data pipelines
• Preferred: experience in high-volume or big data environments, strong problem-solving and analytical skills, ability to work independently in a fast-paced setting

Required skills

• Strong Python and extensive hands-on PySpark experience
• Jupyter Notebooks, SQL and NoSQL databases
• Git for version control and CI/CD; Apache Airflow and/or Jenkins
• Spark performance tuning and clean, testable Python code

What we offer

• Competitive salary package
• Work on production-scale data platforms with modern data engineering tools
• Dubai-based role in a dynamic, collaborative environment