Databricks Data/ETL Developer
Shaik Shoyab aktharUnited StatesSENIORPosted 11m ago
DatabricksPySparkSQLETLELTDatabricks Auto LoaderStructured StreamingDelta LakeBronze-Silver-Gold ArchitectureData ModelingApache SparkAzure Data Lake StorageIAMRow-Level SecurityGitHubCI/CDFivetran
About the Role
We are looking for a Databricks Data/ETL Developer to join our team at Deloitte. This role focuses on designing, developing, and maintaining scalable data engineering pipelines using Databricks, PySpark, and SQL.
Key Responsibilities
• Design, develop, and maintain scalable data engineering pipelines using Databricks, PySpark, and SQL
• Build and support ETL/ELT pipelines for Medicaid Data Warehouse solutions
• Develop streaming pipelines using Databricks Auto Loader and Structured Streaming
• Design and implement Delta Lake tables using the Bronze, Silver, and Gold architecture
• Support data refresh, reconciliation, and validation processes to ensure data accuracy
• Configure and maintain Row-Level Security for data access management
• Support CI/CD pipelines and automate daily dataset updates
• Troubleshoot and optimize Spark jobs and Delta Lake performance
• Handle ad hoc technical and functional questions across the Databricks product suite
• Work closely with clients and technical teams to deliver reliable data engineering solutions
Requirements
• Hands-on experience with Databricks Data Engineering
• Medicaid Data Warehouse (DWH) experience is mandatory
• Strong Medical/Medicaid Eligibility domain experience
• Federal Government or New York State (NYS) project experience is mandatory
• Strong customer-facing communication and problem-solving skills
• Strong experience with Databricks, PySpark, and SQL
• Experience developing ETL/ELT data engineering pipelines
• Hands-on experience with Databricks Auto Loader and Structured Streaming
• Strong understanding of Delta Lake and Bronze-Silver-Gold (Medallion) Architecture
• Experience with data modeling and Delta Lake table design
• Knowledge of Apache Spark performance tuning and optimization
• Experience with Azure cloud services, including Azure Data Lake Storage (ADLS) and IAM
• Experience implementing Row-Level Security (RLS)
• GitHub and CI/CD pipeline experience is a plus
• Fivetran experience is preferred