Jobs → Data & Analytics → Big Data Developer

About the Role We are looking for a Big Data Developer to join our team in Mississauga, ON. This role focuses on developing and maintaining scalable data pipelines using PySpark and Spark SQL. Key Responsibilities • Develop and maintain scalable data pipelines using PySpark and Spark SQL • Write clean, reusable, and optimized Python code for data manipulation, analysis, and automation • Design and implement ETL workflows to process structured and unstructured data • Collaborate with data engineers, analysts, and stakeholders to understand requirements and deliver effective solutions • Optimize Spark jobs for performance, resource utilization, and execution time • Process and analyze large datasets using distributed computing frameworks in cloud or on-premises environments • Develop and maintain unit tests to ensure data accuracy and pipeline reliability • Use Spark SQL to query and manage large datasets stored in Hadoop or cloud storage • Monitor and troubleshoot data pipeline issues to ensure reliable data delivery Requirements • Strong experience with PySpark and Spark SQL • Proficiency in Python programming • Experience with ETL workflows and data pipeline development • Knowledge of distributed computing and big data processing • Experience with Hadoop or cloud-based data storage • Strong debugging, testing, and performance optimization skills