Lead Data Engineer
Gopichand MaddamUnited StatesLEADPosted 7h ago
AzureGCPPythonPySparkSQLETLELTAzure Data FactoryMicrosoft FabricDatabricksAzure Data LakeBigQueryDataflowDataprocCloud StoragePub/SubGitHubCI/CDInfrastructure as CodeDevOpsMLOps
About the Role
We are looking for an experienced Lead Data Engineer with strong hands-on expertise in Azure, GCP, Python, PySpark, SQL, and ETL/ELT to design, develop, and maintain scalable data pipelines and cloud-based data solutions.
Key Responsibilities
• Design and develop scalable batch and streaming data pipelines
• Manage end-to-end data ingestion, transformation, modeling, and orchestration
• Build reusable data engineering frameworks
• Support enterprise analytics and machine learning workloads
• Implement automated testing, CI/CD pipelines, and deployment processes
• Provide technical leadership, mentoring, and code reviews
Requirements
• 10+ years of relevant Data Engineering experience
• Strong expertise in SQL, Python, and PySpark
• Hands-on experience with Azure and GCP (Both Mandatory)
• Experience with Azure Data Factory (ADF), Microsoft Fabric, Databricks, Azure Data Lake
• Experience with GCP BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub
• Experience with ETL/ELT pipelines, data modeling, and distributed data processing
• Experience with GitHub, CI/CD, automated testing, and Infrastructure as Code (IaC)
• Experience with DevOps/MLOps, performance tuning, and production support