Azure Data Engineer
Kelly WilliamsonUnited KingdomMIDPosted 1h ago
Azure Data FactoryAzure DatabricksPySparkSparkSQLAzure Data Lake Storage Gen2Medallion ArchitectureETLELTAzure DevOpsGitHubCI/CDAzure securityTerraformAPI integrationsData governanceMicrosoft Purview
About the Role
We are looking for an experienced, hands-on Azure Data Engineer to join our growing team. This role involves taking ownership of technical delivery and design, building and supporting data ingestion and transformation pipelines within a secure Azure environment.
Key Responsibilities
• Build and support data ingestion and transformation pipelines in Azure
• Design and develop data ingestion pipelines using Azure Data Factory
• Develop scalable transformations using Azure Databricks and PySpark, Spark, and SQL
• Work with Azure Data Lake Storage Gen2 for secure data ingestion, storage, and processing
• Implement Medallion Architecture for ingestion, transformation, and data quality
• Develop ETL/ELT processes integrating APIs, databases, and file-based data sources
• Ensure data quality and validation through reconciliation and exception handling
• Utilize Azure DevOps, GitHub, and CI/CD for version control and automated deployments
• Implement Azure security measures including managed identities, RBAC, and Key Vault
• Troubleshoot pipeline failures and support production workloads
Requirements
• Strong, hands-on Microsoft Azure data engineering experience
• Experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage Gen2
• Proficiency in Python and SQL for development, optimization, and troubleshooting
• Knowledge of Medallion Architecture and ETL/ELT development
• Understanding of data quality and validation processes
• Experience with Azure DevOps, GitHub, and CI/CD
• Familiarity with Azure security practices
• Ability to troubleshoot pipeline failures and support production workloads