Senior Data Engineer / Databricks
Abhishek DevdharUnited StatesSENIORPosted 6d ago
DatabricksPySparkDelta LakeUnity CatalogMedallion ArchitectureSQLStructured StreamingTerraformDatabricks Asset BundlesCloud Platform
About the Role
We are looking for a Senior Data Engineer / Databricks to join our team. This role focuses on leading the design, development, and optimization of scalable, high-performance data solutions.
Key Responsibilities
• Lead data architecture and solution design using Databricks, Delta Lake, Unity Catalog, and Medallion Architecture
• Own end-to-end delivery of data engineering initiatives and provide technical leadership
• Design, develop, and optimize scalable data pipelines using PySpark and SQL
• Build real-time and streaming data solutions using Structured Streaming and/or Delta Live Tables
• Improve platform performance, workload efficiency, and cloud compute cost optimization
• Establish best practices for CI/CD, automated testing, monitoring, observability, and data governance
• Collaborate with business stakeholders and cross-functional teams to translate requirements into technical solutions
• Mentor engineers, support Agile/Scrum delivery, and troubleshoot complex production issues
Requirements
• 10+ years of experience in Data Engineering, including 5+ years of hands-on Databricks experience in production environments
• Strong proficiency in PySpark and SQL
• Experience with real-time data processing and streaming solutions
• Hands-on experience with Unity Catalog, Terraform and/or Databricks Asset Bundles
• Experience with at least one major cloud platform
• Proven technical leadership and team mentoring experience
• Strong communication, stakeholder management, and Agile/Scrum skills