Jobs → Data & Analytics → Lead Data Engineer

About the Role We are looking for an experienced Lead Data Engineer to join an exciting data engineering Proof of Concept (POC). This role requires strong hands-on expertise in Azure Databricks, Python, PySpark, and SQL. Key Responsibilities • Design and implement data solutions using Azure Databricks and PySpark • Develop and optimize complex SQL Stored Procedures • Manage data storage and processing with Delta Lake, Azure Data Factory, and ADLS Gen2 • Optimize Spark performance and manage data partitioning, sharding, and skew • Implement CI/CD pipelines and version control systems Requirements • Advanced Python, Pandas & PySpark skills • Strong T-SQL and experience with complex SQL Stored Procedures • Expertise in Azure Databricks & Delta Lake • Experience with Azure Data Factory (ADF) and ADLS Gen2 • Knowledge of Spark Performance Optimization • Understanding of Partitioning, Sharding & Data Skew Management • CI/CD and Version Control experience