Data Engineer
Shoyab MahfoozRemoteMIDPosted 10d ago
SnowflakeMatillionSQLETLData WarehousingHadoopSparkPythonBigQueryData ModelingData Architecture
About the Role
We are looking for a skilled Data Engineer to join our team. The ideal candidate will have experience in building and optimizing data pipelines, architectures, and data sets, ensuring data availability and enabling data-driven decision-making processes.
Key Responsibilities
• Design, construct, test, and maintain highly scalable data management systems
• Develop and implement architectures that support the extraction, transformation, and loading (ETL) of data from various sources
• Integrate structured and unstructured data from multiple data sources into a unified data system, ensuring data quality and consistency
• Build and maintain data warehouses and data lakes to store and retrieve vast amounts of data efficiently
• Implement data processing frameworks like Hadoop and Spark to process large datasets in real-time or batch processing
• Automate manual processes, optimize data delivery, and develop data monitoring systems to ensure data integrity and accuracy
• Work closely with data scientists, analysts, and other stakeholders to understand data needs and provide technical solutions
• Ensure data governance policies are followed, including data security, data privacy, and compliance with regulations
• Optimize the performance of ETL processes, databases, and data pipelines to handle large volumes of data and reduce processing times
Requirements
• Bachelor’s degree in computer science, Engineering, Information Technology, or a related field
• 5+ years of experience as a Data Engineer or in a similar role
• Proficiency in Snowflake SQL and experience with relational databases (e.g., MySQL, PostgreSQL)
• Knowledge of data warehousing solutions like BigQuery
• Hands-on programming languages like Python
• Strong understanding of Matillion ETL tool and processes
• Experience with data modeling and data architecture design