Lead Azure Data Engineer at rackspace

About the Role We are looking for an experienced and strategic Lead Azure Data Engineer to lead design, development, deployment, and optimization of enterprise data ecosystems on the Microsoft Cloud. You’ll be responsible for building scalable, reliable, and high-performing data platforms and pipelines that support both batch and real-time data processing, analytics, and reporting. Key Responsibilities • Data Platforms & Pipeline Engineering • Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric. • Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines. • Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data. • Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.Microsoft Fabric & Azure Ecosystem • Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics. • Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments. • Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.Databricks Engineering & Dashboards • Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders. • Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution). • Oversee governance, access controls, metadata management, and lineage using Unity Catalog. Leadership & Collaboration • Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality. • Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions. • Partner with stakeholders to define data strategy, standards, and architectural roadmaps.Quality, Governance & Compliance • Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance. • Implement secure, compliant data access and lineage frameworks across cloud data platforms.DevOps & CI/CD • Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows. Required Skills & Qualifications • 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric. • Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming. • Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage. • Solid knowledge of Unity Catalog for data governance, security, and access management. • Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning. • Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions. • Excellent leadership, cross-team collaboration, and communication skills. Good-to-Have Skills • Experience with DBT (Data Build Tool) for modular transformation workflows. • Hands-on expertise in Power BI for analytics and visualization. • Familiarity with data governance tools such as Azure Purview or equivalent. • Certifications like Microsoft Azure Data Engineer (DP-203), Databricks Data Engineer, or Microsoft Fabric Analytics Engineer.