Senior Data Engineer - Python and Terraform skills at Shifttechnology

Shifttechnology

France - Paris·6h ago

SeniorFull-timePythonRustC#AzureTerraformAIMLDevOps

Salary Range

$79,560 - $107,640 /year

Estimated

This salary is estimated based on similar roles. The actual salary may vary.

About the Role

We are looking for a Data Engineer to join our Engineering team at Shift Technology, the leading AI platform for insurance. This role involves driving data architecture design and implementing robust ETL/ELT pipelines to transform insurance data.

Key Responsibilities

→Drive end-to-end data architecture design for scalable and resilient systems
→Design, implement, and maintain robust ETL/ELT pipelines
→Own and optimize Apache Spark-based pipelines
→Translate business requirements into scalable data models and warehouse schemas
→Improve pipeline performance, reliability, scalability, and cost efficiency
→Lead or contribute to POCs and new technology adoption initiatives
→Ensure best practices in data governance, quality, observability, and monitoring
→Work closely with cross-functional teams
→Mentor and support junior engineers
→Take ownership of infrastructure provisioning
→Actively document and diagram technical design and architecture

Requirements

✓8+ years experience, Tech Lead experience is a plus
✓Drive software architecture and understand data model/ETL mapping
✓Experience with Apache Spark, debugging, hardening, and optimizing pipelines
✓3+ years of SQL database and CosmosDB experience
✓3+ years experience in custom ETL design, implementation, and maintenance
✓3+ years experience with Data Modeling
✓Strong understanding of distributed data systems, data warehousing concepts, batch and event-driven processing, and performance and scalability troubleshooting
✓Hands-on experience with Databricks, DBT, Infrastructure as Code (Terraform, ARM, Bicep, etc.), and Azure data services
✓Experience provisioning infrastructure and setting up data pipelines independently