Loading...
SeniorFull-timePythonAWSKubernetesMongoDBElasticsearchGitData
Salary Range
$55 - $60 /month
About the Role
We are looking for a Data Engineer to join our team at Brillio, a leading digital technology service provider. This role focuses on providing production support for data engineering applications and ensuring high availability of data pipelines.
Key Responsibilities
- →Provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents
- →Monitor, troubleshoot, and optimize PySpark-based data processing jobs
- →Support and maintain data processing applications deployed on AWS services, including EKS and EMR
- →Manage and orchestrate workflows using Control-M, including scheduling, monitoring, and error handling
- →Perform root cause analysis and implement permanent fixes for recurring production issues
- →Oversee data storage and management in MongoDB and Snowflake, ensuring data quality and consistency
- →Monitor and analyze application and system logs using Elasticsearch Service (ESS) to proactively identify and resolve issues
- →Collaborate with data engineering, analytics, and operations teams to deliver reliable data solutions
- →Maintain and update technical documentation related to production support processes and incident management
- →Participate in on-call rotations and respond to critical production alerts as needed
Requirements
- ✓3+ years of hands-on experience in production support for data engineering or analytics applications
- ✓Proficiency in PySpark for managing and troubleshooting data processing jobs
- ✓Solid experience with AWS cloud services, including EKS (Elastic Kubernetes Service) and EMR (Elastic MapReduce), S3, Glue, Step Function
- ✓Hands-on experience with Control-M for workflow orchestration and monitoring
- ✓Working knowledge of MongoDB and Snowflake for production data management
- ✓Experience with log monitoring and analysis using Elasticsearch Service (ESS)
- ✓Good understanding of ETL processes, incident management, and production monitoring best practices
- ✓Strong problem-solving, troubleshooting, and analytical skills
- ✓Excellent communication and collaboration abilities
Free to sign up. Takes 30 seconds.
