GCP Data Engineer | BPCE-SI at NatixisInPortugal

Natixis in Portugal is currently seeking a Data Engineer to join its BPCE SI (BPCE Solutions Informatiques) team, within the IT (Information Technologies) department, to work in a hybrid model in the Porto or Lisbon office.  Responsibilities    Data and Database Migration: • Actively participate in the design and execution of the data and ETL process migration strategy from SQL Server to GCP. • Develop and optimize scripts and pipelines for efficiently transferring and transforming data into BigQuery.Design and Development of Data Pipelines on GCP: • Design, develop, and maintain robust and high-performing ETL/ELT pipelines on GCP, replacing or rebuilding existing flows. • Orchestrate and schedule these pipelines using Google Cloud Composer (or Apache Airflow) to ensure reliable migration and operationalization.Database Management and Optimization on GCP: • Develop and optimize complex SQL queries for BigQuery, taking into account the specificities of migrating from SQL Server to BigQuery (e.g., optimizing partitioning, clustering). • Contribute to the design and implementation of optimized data schemas for BigQuery, adapted to new cloud architectures.Development and Automation: • Use Python to develop scripts for automating migration tasks, data transformation, and integration with other GCP services. • Implement processes to ensure data quality and integrity during and after migration.Collaboration and Support: • Collaborate closely with current teams (SQL Server administrators, analysts) to understand the existing environment and facilitate the transition. • Exchange with end-users (business lines, data scientists) to understand their needs and ensure the new platform meets their expectations. • Participate in code reviews, development best practices, and testing strategies for the new platform. • Ensure monitoring, debugging, and incident resolution related to data pipelines during the transition phase and upon production deployment.Technological Watch and Optimization: • Stay informed about best practices for migration and the latest developments on Google Cloud Platform. • Propose improvements and optimizations for the new Data platform on GCP.