Jobs → Data & Analytics → Data Engineer

About the Role We are seeking a Data Engineer to join our Middle Office team. This role involves designing and implementing scalable data solutions to support business processes, analytics, and reporting. Key Responsibilities • Onboard new datasets to support Middle Office business processes, analytics, reporting, and downstream applications • Work with business and technology stakeholders to understand data requirements, source systems, data structures, business rules, and downstream consumption patterns • Design scalable data ingestion, transformation, and processing solutions using AWS and Apache Spark/PySpark • Design and work with large-scale data lake tables using technologies such as Apache Hudi and Apache Iceberg • Build workflow orchestration and scheduling solutions using Apache Airflow or equivalent orchestration technologies • Perform data modeling and design appropriate schemas, partitioning strategies, data structures, and transformation layers for analytical and operational use cases Requirements • Strong hands-on experience with AWS data and analytics services, particularly AWS EMR, AWS Glue and Glue Data Catalog, Amazon Athena, Amazon S3, AWS IAM, and general AWS security concepts • Strong experience working with modern data lake/table formats, particularly Apache Hudi and Apache Iceberg • Understanding of concepts such as schema evolution, partitioning, incremental processing, upserts, snapshots, and management of large analytical tables • Strong programming skills in Python and PySpark • Experience in object-oriented and modular design, unit and integration testing, error and exception handling, logging, code reviews, and Git-based development workflows • Strong hands-on experience with Apache Airflow or a comparable enterprise orchestration platform • Experience designing DAGs and workflow dependencies, scheduling, retry and failure-handling strategies, backfill and rerun processes, monitoring and alerting, parameterized and reusable workflows • Experience with Amazon MWAA is an asset • Strong understanding of data modeling, data pipeline architecture, data lake architecture, schema design, data partitioning, incremental data processing, historical data management, data quality and reconciliation, and data lineage • Capable of translating business requirements into a scalable technical data model and an operationally supportable data pipeline • Working knowledge of modern DevOps and infrastructure-as-code practices, including Terraform, GitHub Actions, Git, CI/CD pipelines, automated testing and deployment, environment configuration and promotion