Data Engineer
SRINIVAS MRemoteSENIORPosted 2h ago
AWSSnowflakeIICSPySparkEKSEMREC2AWS Step FunctionsControl-MAWS Glue Data CatalogDynamoDB
About the Role
We are seeking a Data Engineer to design and implement end-to-end data pipelines using various cloud technologies.
Key Responsibilities
• Architect end-to-end pipelines using AWS, Snowflake, IICS, and PySpark
• Design and deploy containerized PySpark workloads on EKS
• Build and tune EMR clusters on EC2 or EKS for large-scale processing
• Orchestrate workflows using AWS Step Functions and Control-M
• Manage metadata using AWS Glue Data Catalog and integrate it with Snowflake Iceberg
• Use DynamoDB for pipeline state, checkpoints, and configuration
• Lead implementation of Snowflake capabilities including Iceberg Tables, Dynamic Tables, Views, Materialized Views, Tasks and Streams, Data Governance, Horizon Catalog, Semantic Views, Cortex (Cortex AI), Snowflake Agents, and token-usage optimization strategies
Requirements
• 9+ years of experience in data engineering
• Strong experience with AWS, Snowflake, and PySpark
• Experience with containerized workloads and EKS
• Knowledge of EMR clusters and large-scale processing
• Familiarity with workflow orchestration tools
• Experience with metadata management and integration
• Understanding of NoSQL databases like DynamoDB
• Leadership experience in implementing cloud data solutions