Jobs → Data & Analytics → Lead Data Engineer

About the Role We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, Scala, Apache Spark, Hadoop, and Control-M. The ideal candidate will be responsible for developing, optimizing, and maintaining scalable data pipelines and distributed data processing solutions. Key Responsibilities • Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, Scala, and Apache Spark. • Build and optimize ETL/ELT workflows for large-scale data processing. • Work with Hadoop ecosystem technologies for distributed data storage and processing. • Schedule, monitor, and troubleshoot batch jobs and workflows using Control-M. • Optimize Spark jobs and Databricks workloads for performance and scalability. • Perform data transformation, validation, quality checks, and troubleshooting. • Collaborate with data architects, analysts, and business teams to deliver reliable data solutions. Requirements • Strong hands-on experience with Databricks, Python, PySpark, Scala, and Apache Spark. • Experience with the Hadoop ecosystem and distributed data processing. • Hands-on experience with Control-M job scheduling, monitoring, and dependency management. • Strong understanding of ETL/ELT processes, data engineering concepts, and batch processing. • Proficiency in SQL and performance tuning. • Strong problem-solving and debugging skills.