Lead Data Engineer
Teja MUnited StatesLEADPosted 1h ago
DatabricksPySparkSpark SQLDelta LakePythonSQLETLELTAzureAWSGCPADFAirflowData ModelingData WarehousingGitCI/CDDevOpsDelta Live TablesUnity CatalogStructured Streaming
About the Role
We are seeking a Lead Data Engineer with extensive experience in Data Engineering and hands-on expertise in Databricks. This role involves managing customer issues, building scalable data solutions, and collaborating with cross-functional teams.
Key Responsibilities
• Design and implement scalable ETL/ELT pipelines and data engineering solutions
• Manage customer escalations and resolve issues effectively
• Develop and maintain data models and data warehousing solutions
• Optimize data performance and implement best practices
• Collaborate with business and technical teams to deliver enterprise data solutions
Requirements
• 14+ years of IT experience with 10+ years in Data Engineering
• Mandatory hands-on experience in Databricks (PySpark, Spark SQL, Delta Lake)
• Strong programming skills in Python and SQL
• Experience with Azure/AWS/GCP, preferably Azure, and orchestration tools (ADF/Airflow)
• Strong understanding of data modeling, data warehousing, and performance optimization
• Experience with Git, CI/CD, and DevOps best practices
• Knowledge of Delta Live Tables, Unity Catalog, and Structured Streaming is preferred
• Excellent analytical, problem-solving, and communication skills