Python, AWS & Databricks Engineer
Kartik TyagiUnited StatesMIDPosted 1h ago
PythonAWSDatabricksPySparkSpark SQLETLELTS3GlueLambdaRedshiftSQLData WarehousingData LakeDelta LakeLakehouseIAMCI/CDGitAgileScrum
About the Role
We are looking for an experienced Python, AWS & Databricks Engineer to join our team at Capital Group in Irvine, CA. The ideal candidate will have strong experience in Python programming, AWS cloud services, and Databricks-based data engineering.
Key Responsibilities
• Design, develop, and maintain scalable data pipelines using Python and Databricks.
• Build and optimize data processing workflows using PySpark and Spark SQL.
• Develop and manage data engineering solutions using AWS services.
• Implement ETL/ELT processes for data ingestion, transformation, and integration.
• Work with AWS S3, Glue, Lambda, Redshift, and other cloud services.
• Develop data transformations and optimize Spark jobs for performance and scalability.
• Implement data quality checks, validation, and error-handling mechanisms.
• Integrate data from multiple sources into centralized data platforms.
• Collaborate with data architects, analysts, and cross-functional teams.
• Monitor, troubleshoot, and optimize data pipelines to ensure reliability.
• Follow best practices for security, governance, CI/CD, and version control.
• Participate in code reviews, technical discussions, and Agile development activities.
Requirements
• Strong hands-on experience with Python programming.
• Extensive experience with AWS cloud services.
• Strong experience with Databricks and Apache Spark.
• Hands-on experience with PySpark and Spark SQL.
• Experience developing ETL/ELT data pipelines.
• Strong SQL skills and experience working with relational databases.
• Experience with AWS S3, Glue, Lambda, and Redshift.
• Understanding of data warehousing and data lake architectures.
• Experience with data integration, transformation, and optimization.
• Strong troubleshooting and problem-solving skills.