Data Engineer
KevellsUnited StatesSENIORPosted 31m ago
AWSGCPSparkPySparkDatabricksAirflowSQLPythonAmazon S3Amazon RedshiftGoogle Cloud StorageGoogle BigQueryGoogle Cloud DataprocApache HiveTerraformCI/CD
About the Role
We are seeking a Data Engineer to support an enterprise data platform and its migration from AWS to Google Cloud Platform (GCP). This role involves maintaining and optimizing production pipelines in AWS while building and migrating scalable data solutions on GCP.
Key Responsibilities
• Design, build, optimize, and maintain scalable batch and distributed data pipelines.
• Support AWS workloads built with Amazon S3, Amazon Redshift, Databricks, Apache Airflow, and PySpark.
• Migrate data storage and processing workloads from Amazon S3 to Google Cloud Storage (GCS).
• Modernize Redshift data warehouse workloads and SQL for Google BigQuery.
• Develop and tune Spark and PySpark workloads on Google Cloud Dataproc.
• Build and maintain Airflow DAGs, including migration and orchestration improvements.
• Work with Apache Hive tables, partitions, metadata, and Spark SQL.
• Protect data quality, integrity, reliability, and performance throughout migration.
• Troubleshoot pipeline failures, data inconsistencies, and performance bottlenecks.
• Collaborate with engineers, platform teams, architects, and business partners.
Requirements
• 10+ years of overall IT experience, with substantial data engineering experience.
• Strong hands-on development experience with Python, PySpark, Apache Spark, and SQL.
• Strong AWS data engineering experience, particularly with Amazon S3 and Redshift.
• Experience with Databricks and Apache Airflow.
• Hands-on experience with GCP services, including GCS, Dataproc, and BigQuery.
• Experience migrating or modernizing data workloads across cloud platforms.
• Strong understanding of distributed computing, data partitioning, query optimization, and performance tuning.
• Experience with Apache Hive and large-scale data processing.
• Strong debugging, problem-solving, and communication skills.