Databricks Data/ETL Developer
shiva VUnited StatesMIDPosted 28m ago
DatabricksPySparkSQLETLdata engineeringDatabricks AutoloaderStructured StreamingDelta LakeBronze-Silver-Gold Medallion ArchitectureData modelingSpark performance optimizationAzure servicesADLSIAMGitHubCI/CDFiveTran
About the Role
We are looking for a Databricks Data/ETL Developer to join our team in the healthcare domain, specifically focusing on Medicaid data. This role requires expertise in designing, developing, and maintaining data pipelines using Databricks and related technologies.
Key Responsibilities
• Design, develop, and maintain Databricks data pipelines
• Support ETL and CI/CD pipelines for data lake tables
• Manage daily data refresh and reconciliation processes
• Validate data accuracy and implement row-level security
• Automate daily dataset updates
Requirements
• Medicaid Data Warehouse experience is mandatory
• NY State client experience preferred, OR experience working with any other State client
• Strong Healthcare Medicaid Eligibility domain experience
• Hands-on experience with Databricks, PySpark, and SQL
• ETL and data engineering pipeline development
• Databricks Autoloader or Structured Streaming
• Delta Lake and Bronze-Silver-Gold Medallion Architecture
• Data modeling and Spark performance optimization
• Azure services, including ADLS and IAM
• GitHub and CI/CD experience is a plus
• FiveTran experience preferred
• Relevant Databricks certifications are a plus