Jobs → Data & Analytics → Databricks Quality Lead Tester

About the Role We are seeking a Databricks Quality Lead Tester to join a leading healthcare payer’s data quality team. This role involves end-to-end ETL testing and validation of large-scale data pipelines in Databricks. Key Responsibilities • Perform end-to-end ETL testing and validation of large-scale data pipelines in Databricks • Conduct source-to-target mapping, data reconciliation, and integrity checks • Validate PySpark, Spark SQL, and Delta Lake transformations • Perform backend testing with complex SQL queries • Test Databricks and Snowflake environments, including Delta tables, schema changes, incremental loads, and CDC • Validate Snowflake data, Snowpark and Dynamic Tables, and migrations between Databricks and Snowflake • Develop and run automated data quality scripts in Python/PySpark/SQL • Utilize frameworks such as Great Expectations, dbt Tests, Soda, or custom solutions • Integrate automated tests into CI/CD pipelines • Identify, troubleshoot, and report data defects • Support healthcare payer data validation, understanding of Medicare/Medicaid, HL7/FHIR, and PHI/PII compliance Requirements • 7–10 years of ETL/Data Warehouse Testing experience • At least 5 years of Databricks testing expertise • Strong hands-on SQL and PySpark skills • Proficiency with Databricks, Spark SQL, and Delta Lake • Experience in ETL testing, data validation, and source-to-target reconciliation • Python-based test automation experience • Solid defect management and troubleshooting abilities • Knowledge of healthcare payer data preferred • Familiarity with Medicare/Medicaid, claims, HL7/FHIR, PHI/PII privacy requirements • Experience with Snowflake, Snowpark, or Dynamic Tables is a plus • Familiarity with Great Expectations, dbt Tests, or Soda is a plus