Jobs → Data & Analytics → Technology Lead – Big Data

About the Role We are seeking a Technology Lead with expertise in Big Data technologies to join our team. This role involves leading data pipeline development and migration initiatives, focusing on Hadoop and AWS cloud-native architectures. Key Responsibilities • Design, develop, and optimize scalable batch and real-time data pipelines • Manage enterprise data lakehouse architectures across AWS and Cloudera environments • Implement CDC and complex SQL workflows to integrate relational data into dimensional data warehouses • Automate and orchestrate data pipelines using Python and Unix shell scripting • Lead Hadoop-to-AWS cloud migration initiatives • Monitor and optimize Spark jobs and ensure data quality, reliability, and regulatory compliance • Collaborate with data architects, analysts, and platform engineering teams to deliver scalable data solutions Requirements • Strong hands-on experience with AWS and Hadoop • 8–10 years of experience in Big Data engineering and data pipeline development • Expertise in Apache Spark using Scala/Python and Core Java • Hands-on experience with Apache Kafka for real-time data streaming • Experience with AWS S3, Cloudera, Apache Hive, and Apache Iceberg • Strong SQL skills, including CDC and integration with Oracle and SQL Server • Experience with dimensional data warehouse modeling • Proficiency in Python and Unix shell scripting for automation • Experience migrating legacy Hadoop workloads to AWS cloud-native architectures • Knowledge of JVM internals and Spark performance tuning Preferred • Telecom/communications domain experience, including CDRs and network telemetry data