Jobs → Data & Analytics → Lead Data Engineer

About the Role We are seeking a Lead Data Engineer to join our team for a 12-month contract. This role involves designing and developing scalable data pipelines using Azure technologies. Key Responsibilities • Design and develop scalable batch and streaming data pipelines • Build data ingestion and transformation workflows using Azure Databricks and ADF • Optimize Spark jobs and troubleshoot performance issues • Work with ADLS Gen2, Delta Lake, Unity Catalog, Azure Synapse, and Event Hubs • Implement CI/CD pipelines using Azure DevOps or GitHub Actions • Automate infrastructure provisioning using Terraform • Ensure data quality, security, monitoring, and governance Requirements • 8+ years of experience in data engineering • Strong experience with Azure Data Factory and Azure Databricks • Proficiency in PySpark and SQL • Experience with CI/CD pipelines and infrastructure automation Preferred Skills • Scala, Apache Airflow, Kafka, Azure Key Vault, and Azure Monitor