Jobs → Data & Analytics → Azure Databricks Engineer

About the Role We are looking for an experienced Azure Databricks Engineer with strong hands-on expertise in Azure Databricks, PySpark, Python, and SQL. This role focuses on designing, developing, and optimizing scalable data engineering solutions and large-scale data processing workflows using Azure Cloud services. Key Responsibilities • Design, develop, and maintain scalable data pipelines using Azure Databricks • Build and optimize ETL/ELT workflows using Azure Data Factory (ADF) • Develop data processing solutions using PySpark, Python, and Spark SQL • Work with Azure Data Lake Storage (ADLS Gen2) for data ingestion and storage • Implement Delta Lake architecture, including Delta Tables, Delta Live Tables, and data versioning • Develop and optimize complex SQL queries and data transformations • Implement Medallion Architecture (Bronze, Silver, and Gold layers) • Integrate data from multiple sources, including databases, APIs, and cloud platforms • Monitor, troubleshoot, and optimize Spark jobs and Databricks clusters • Implement data security, governance, and access control using Unity Catalog • Work with Azure DevOps and CI/CD pipelines for deployment automation • Collaborate with data architects, business analysts, and cross-functional teams Requirements • Strong hands-on experience with Azure Databricks • Proficiency in PySpark, Python, and Spark SQL • Experience with Azure Data Factory (ADF) • Strong knowledge of Azure Data Lake Storage (ADLS Gen2) • Hands-on experience with Delta Lake and Delta Live Tables • Knowledge of Medallion Architecture • Experience with Unity Catalog and data governance • Familiarity with Azure DevOps and CI/CD pipelines • Strong SQL and data pipeline optimization skills