Loading...
SeniorFull-timePythonAWSGCPTerraformCI/CD
Salary Range
$26,520 - $35,880 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for a Data Engineer to join our team at Capco, a global technology and management consulting firm. This role focuses on building and operationalizing data pipelines, ETL workflows, and analytics platforms.
Key Responsibilities
- →Build scalable ETL/ELT pipelines using PySpark on distributed processing frameworks
- →Orchestrate workflows using Apache Airflow (DAG design, scheduling, monitoring)
- →Develop data ingestion and transformation jobs using AWS Glue
- →Manage secure, compliant data access using AWS Lake Formation
- →Maintain and optimize AWS Glue Data Catalog for metadata, schema, and table management
- →Work with analytics teams to publish datasets for BI and dashboards
- →Build and support visualizations using Amazon QuickSight
- →Ensure data quality, performance, and reliability across all pipelines
Requirements
- ✓Strong hands-on experience with PySpark for large-scale data processing
- ✓Deep knowledge of Airflow DAGs, operators, sensors, and CI/CD integration
- ✓Expertise in AWS Glue (ETL jobs, crawlers, Glue Studio, Glue Job Bookmarks)
- ✓Experience with Lake Formation permissions, governance, and data lakes
- ✓Familiarity with Glue Data Catalog for metadata management
- ✓Ability to build dashboards in Amazon QuickSight
- ✓Understanding of data modeling, partitioning, and performance optimization
Free to sign up. Takes 30 seconds.
