Data Engineer (PySpark)

About the role

We are looking for an experienced Data Engineer with deep expertise in PySpark and the Cloudera Data Platform to design and maintain scalable data pipelines that support our analytics initiatives.

Key responsibilities

• Design, develop, and maintain highly scalable ETL pipelines using PySpark on CDP.
• Implement data ingestion from relational databases, APIs, and file systems into data lakes or warehouses.
• Transform, cleanse, and process large datasets to meet business requirements.
• Optimize PySpark code and CDP components for performance and resource utilization.
• Implement data quality checks, monitoring, and validation routines.
• Automate workflows with orchestration tools such as Apache Oozie or Airflow.
• Monitor pipeline performance, troubleshoot issues, and perform routine maintenance.
• Collaborate with data engineers, analysts, product managers, and other stakeholders.
• Document engineering processes, code, and pipeline configurations.

Required profile

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems or related field.
• 8+ years of experience as a Data Engineer with a strong focus on PySpark and CDP.

Required skills

• Advanced proficiency in PySpark (RDDs, DataFrames, optimization).
• Strong experience with Cloudera Data Platform components (Cloudera Manager, Hive, Impala, HDFS, HBase).
• Knowledge of data warehousing concepts and SQL‑based tools (Hive, Impala).
• Familiarity with Hadoop ecosystem, Kafka, and distributed computing.
• Experience with orchestration and scheduling tools such as Apache Oozie or Airflow.