Jobs → Data & Analytics → Lead Data Engineer

About the Role We are seeking a Lead Data Engineer to design and develop cloud-native data pipelines on AWS and Snowflake. This role involves driving advanced capabilities and AI proofs of concept, while mentoring mid-level developers and collaborating with stakeholders. Key Responsibilities • Architect end-to-end pipelines using AWS, Snowflake, IICS, and PySpark • Design and deploy containerized PySpark workloads on EKS • Build and tune EMR clusters on EC2 or EKS for large-scale processing • Orchestrate workflows using AWS Step Functions and Control-M • Manage metadata using AWS Glue Data Catalog and integrate it with Snowflake Iceberg • Use DynamoDB for pipeline state, checkpoints, and configuration • Lead implementation of Snowflake capabilities, including Iceberg Tables, Dynamic Tables, Views, Materialized Views, Tasks and Streams, Data Governance, Horizon Catalog, Semantic Views, Cortex (CoCo), Snowflake Agents, and token-usage optimization strategies • Build RAG and Agentic AI PoCs covering vector search, BM25 hybrid retrieval, evaluation frameworks, human-in-the-loop controls, and security controls • Develop PySpark and Snowpark pipelines for ingestion and transformation • Lead IICS file-based ingestion design • Conduct code reviews, improve performance, and mentor engineers Requirements • Strong hands-on experience with AWS data services, including EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, and S3 • Advanced Snowflake expertise • Expertise in PySpark and Snowpark • Hands-on experience with IICS for file-based ETL ingestion • Experience building RAG and Agentic AI PoCs • Strong SQL and data modeling skills • Leadership and mentoring capability • Knowledge of Control-M orchestration