Lead Data Engineer
Sourav D.(SD)United StatesLEADPosted 2h ago
DatabricksDelta LakeUnity CatalogPySparkPythonSQLAWS S3Data Pipeline OptimizationCI/CDAuto LoaderStructured StreamingCDCAirflow
About the Role
We are looking for a hands-on Lead Data Engineer with strong expertise in Databricks, PySpark, Python, and SQL to build and optimize scalable data pipelines.
Key Responsibilities
• Build and optimize scalable data pipelines
• Implement Medallion Architecture (Bronze/Silver/Gold)
• Work with AWS S3 and optimize data pipelines
• Implement CI/CD processes
Requirements
• 9-15 years of experience in data engineering
• Strong expertise in Databricks, PySpark, Python, and SQL
• Experience with Delta Lake and Unity Catalog
• Knowledge of Medallion Architecture
• Experience with AWS S3 and data pipeline optimization
• Experience with CI/CD
• Technical leadership and code reviews
Experience with Auto Loader, Structured Streaming, CDC, or Airflow is a plus.