Databricks Data Engineer
rishabh pandeyUnited StatesSENIORPosted 1h ago
DatabricksDelta LakeUnity CatalogDelta Live TablesPythonPySparkData PipelinesETLELTMedallion ArchitectureBronze Silver GoldData IngestionEDIData QualityData ValidationData ReconciliationException HandlingData GovernancePipeline MonitoringBig DataLakehouseSparkProduction Support
About the Role
We are looking for a hands-on Databricks Data Engineer to develop and maintain data pipelines for an EDI Proof of Concept (POC).
Key Responsibilities
• Build and maintain data ingestion pipelines using Databricks.
• Work on Bronze, Silver, and Gold data architecture.
• Develop data transformations using Python and PySpark.
• Implement Delta Live Tables (DLT) and data quality validations.
• Handle data exceptions, reconciliation, and error management.
• Work with external file feeds and high-volume data processing.
• Implement pipeline monitoring, auditing, and data validation.
• Ensure data accuracy, reliability, and performance.
Requirements
• 8+ years of experience with Databricks, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Python, and PySpark.