Jobs → Engineering → PySpark Developer

About the Role We are looking for a skilled PySpark Developer to design, build, and optimize large-scale data processing pipelines using Python, Apache Spark, SQL, and Azure Databricks. This role focuses on building reliable, scalable, and high-performance data solutions in the cloud. Key Responsibilities • Develop, test, and maintain ETL/ELT pipelines using PySpark and Spark SQL • Build ingestion and transformation workflows in Azure Databricks • Process structured, semi-structured, and unstructured data at scale • Implement Delta Lake solutions with Azure Data Lake Storage Gen2 • Optimize Spark jobs with partitioning, caching, and broadcast joins • Implement data quality checks, CDC processing, and reconciliation • Collaborate with architects, analysts, QA, and business stakeholders • Use Git, Azure DevOps, or GitHub Actions for CI/CD • Troubleshoot pipeline failures and improve reliability Requirements • Strong hands-on experience with Python and PySpark • Proficiency in Apache Spark architecture, Spark SQL, DataFrames • Strong SQL skills (joins, window functions, query optimization) • Experience developing pipelines in Azure Databricks • Knowledge of Delta Lake, lakehouse architecture, and ADLS Gen2 • Familiarity with batch processing, incremental loads, and data integration • Experience with Git, CI/CD, and Agile practices • Strong debugging and problem-solving skills Preferred Qualifications • Experience with Azure Data Factory, Synapse Analytics • Knowledge of Delta Lake optimization (OPTIMIZE, partitioning, compaction) • Exposure to Databricks Workflows, Unity Catalog, secure data access • Familiarity with Kafka, Event Hubs, streaming data • Experience with dimensional modeling and data warehousing Experience • 5+ years in data engineering, with 3+ years of hands-on PySpark development • Strong preference for Azure Databricks experience