Databricks Data/ETL Developer
Divyanshu (.RemoteMIDPosted 3h ago
DatabricksPySparkSpark SQLStreamingDelta LakeAzureADLS Gen2IAMFivetranGitHub CI/CDRow-level security
About the Role
We are actively seeking an experienced Databricks Data/ETL Developer with strong customer-facing presence and deep expertise in enterprise data engineering on Azure Databricks.
Key Responsibilities
• Design, build, and maintain scalable data pipelines in Databricks using PySpark or Spark SQL
• Build streaming data ingestion pipelines using Databricks Auto Loader and Structured Streaming
• Design and implement Delta Lake tables across Bronze, Silver, and Gold layers
• Implement Spark and Delta Lake performance tuning (partitioning, caching, Z-Ordering, file compaction)
• Leverage Azure services including ADLS Gen2 and Azure IAM for secure data access
Requirements
• Ex-Deloitte consulting OR Direct Medicaid experience (Must Have)
• Strong expertise in Databricks and PySpark
• Experience with streaming architectures and data ingestion
• Knowledge of Medallion Architecture and Delta Lake
• Understanding of cloud infrastructure and data security
Benefits
• Opportunity to work on challenging data engineering projects
• Collaborative and innovative work environment