Jobs → Data & Analytics → Lead Data Platform Engineer

About the Role We are looking for an experienced Lead Data Platform Engineer to drive the design, build, and optimization of our enterprise data platform. This role will lead technical strategy, guide engineering best practices, and work closely with cross-functional teams to deliver scalable, secure, and high-performing data solutions across Hadoop, Hive, and Azure—including Databricks. Key Responsibilities • Lead the architecture, development, and enhancement of large-scale data platforms supporting analytics, machine learning, and operational workloads. • Design and optimize big data solutions leveraging Hadoop ecosystem technologies such as HDFS, Hive, Spark, YARN, and related components. • Develop and manage data pipelines and transformations using Azure Data services (Data Lake Storage, Data Factory, Synapse, Databricks). • Implement and enforce robust data governance, security, and quality frameworks across all data layers. • Partner with data engineering, analytics, product, and infrastructure teams to translate business needs into scalable technical designs. • Drive performance tuning, capacity planning, and cost optimization across both on-premise and cloud data platforms. • Mentor and technically guide data engineers; establish engineering standards, reusable patterns, and best practices. • Oversee CI/CD practices, deployment, and monitoring for data workflows. • Evaluate emerging technologies and contribute to long-term platform strategy and modernization initiatives. Required Qualifications • 8+ years of experience in data engineering or data platform roles, with 3+ years in a technical lead or architect capacity. • Strong hands-on experience with Hadoop ecosystem components (HDFS, Hive, Spark, Oozie, Ranger, Airflow etc.). • Deep expertise with Azure data services (Azure Data Lake Storage, Azure Data Factory, Azure Synapse, Azure Functions, Key Vault). • Advanced experience building solutions in Databricks, including Spark optimization, Delta Lake, and Unity Catalog. • Proficiency in Python, SQL, and distributed data processing frameworks. • Experience with DevOps practices, CI/CD pipelines, and Infrastructure-as-Code (e.g., Terraform or ARM). • Strong understanding of data modeling, storage formats (Parquet, ORC, Delta), and data governance frameworks. • Proven ability to lead technical teams, communicate clearly, and influence architecture decisions.