Jobs → Data & Analytics → Lead Data Engineer – Databricks

About the Role We are seeking a Lead Data Engineer with expertise in Databricks to join our team. This role involves designing, building, and optimizing scalable data solutions for the insurance industry. Key Responsibilities • Design, build and optimize scalable batch and streaming ETL/ELT pipelines on Databricks using PySpark, Spark SQL, Delta Lake, and other technologies. • Architect and implement Lakehouse solutions using the Medallion pattern with Unity Catalog for governance. • Build and maintain enterprise data warehouse models on various cloud platforms. • Migrate legacy ETL and on-prem warehouses to modern cloud lakehouse platforms. • Enable BI and reporting by delivering semantic layers and curated datasets for Power BI and other BI tools. • Integrate data from various sources using tools like ADF, AWS Glue, and Kafka. • Implement data quality, observability, and testing frameworks. • Apply CI/CD and DataOps practices for efficient data management. • Optimize cost and performance across clusters and storage. • Partner with architects, analysts, and business stakeholders; mentor junior engineers. Requirements • 8–12 years of data engineering experience with large-scale ETL and data warehouse solutions. • Strong hands-on Databricks expertise: PySpark, Spark SQL, Delta Lake, Unity Catalog, Workflows/Jobs, DLT/Lakeflow, Databricks SQL, performance tuning. • Advanced SQL and Python; Scala is a plus. • Proven data warehousing and data modeling experience (Kimball dimensional modeling, SCDs, fact/dimension design). • Hands-on experience with at least one hyperscaler, ideally multi-cloud: Azure, AWS, or GCP. • BI tools: strong Power BI (data modeling, DAX, dataset optimization); exposure to Tableau, Looker, or Qlik. • Experience with orchestration tools like Databricks Workflows, Airflow, and ADF, and modern tools such as dbt. • Solid understanding of data governance, security, lineage, and metadata management. • Experience with CI/CD, Git-based workflows, and Agile delivery. • Excellent communication skills and comfort working directly with client stakeholders. Good to Have • Insurance domain experience (P&C, Specialty/E&S, brokerage, MGAs): policy, claims, billing, underwriting, loss runs, bordereaux, exposure and reinsurance data. • Familiarity with insurance platforms and data standards (Guidewire, Duck Creek, Majesco, Applied Epic, ACORD). • Databricks certifications: Data Engineer Associate / Professional. • Cloud certifications (Azure Data Engineer, AWS Data Engineer, GCP Professional Data Engineer). • Snowflake and/or Microsoft Fabric experience. • Exposure to ML/AI data pipelines, feature stores, MLflow, or GenAI/RAG data preparation. • Experience with real-time streaming (Structured Streaming, Kafka).