Jobs → Engineering → Lead PySpark Developer

About the Role We are looking for an experienced Lead PySpark Developer to join a leading global IT services and digital engineering organization, working on a large-scale financial services and market surveillance platform. Key Responsibilities • Design, develop, and optimize scalable, high-performance PySpark data pipelines • Lead development activities, conduct code reviews, and mentor junior developers • Build complex data transformations, calculations, and aggregations using Python, PySpark, Pandas, and NumPy • Optimize Spark applications using window functions, salting, column pruning, and performance-tuning techniques • Design concurrent data-processing architectures with robust error handling • Translate business requirements into technical specifications and implement reliable solutions • Troubleshoot application issues and ensure adherence to coding standards and best practices Requirements • 15+ years of overall IT experience with strong hands-on Python and PySpark development • Expertise in Spark architecture, performance optimization, and window functions • Strong Core Python skills, including OOP, classes, decorators, lambda functions, exception handling, and parallel processing • Extensive experience with Pandas, NumPy, vectorization, data wrangling, and aggregations • Strong understanding of data pipelines, database connectivity, data loading, and transformations • Proven experience in development team leadership, solution design, and code reviews • Good understanding of Generative AI • Experience with Git, debugging, and application integration Good to Have • Experience with Django, PostgreSQL, and Apache Airflow • Knowledge of financial markets, asset classes, and trading instruments • Exposure to market surveillance, financial regulations, and compliance frameworks such as MAR, MiFID, and Dodd-Frank