Job Description
We are looking for a skilled Data Engineer with strong experience in big data technologies, distributed processing, data pipelines and cloud-based data platforms. The candidate should have hands-on expertise in PySpark, Python, Databricks, Spark, Scala and Snowflake, along with experience in ETL/ELT development and data quality management.
Key Responsibilities
Design, develop and maintain scalable data pipelines using PySpark, Python and Spark.
Develop and optimise ETL/ELT workflows for structured and semi-structured data.
Work with Databricks and Snowflake for data processing and storage.
Develop data transformation and integration solutions using distributed computing technologies.
Work with Scala and Python for data engineering tasks.
Process data in formats such as XML, JSON, Avro and Parquet.
Implement data validation, reconciliation and data quality checks.
Design and maintain data warehouse solutions.
Optimise Spark jobs and troubleshoot pipeline performance issues.
Handle large datasets and ensure data accuracy, consistency and reliability.
Collaborate with data analysts, architects and business teams.
Maintain technical documentation and follow data engineering best practices.
Support production data pipelines and resolve operational issues.