Jobs → Data & Analytics → Lead Data Engineer / Data Solution Architect

About the Role Infinity Arc Technologies is seeking an experienced Lead Data Engineer / Data Solution Architect to join a high-impact data modernization initiative focused on optimizing complex data processing workloads. Key Responsibilities • Refactor complex SQL Stored Procedures into highly optimized, vectorized Python code • Develop and optimize solutions within Azure Databricks using PySpark • Design scalable data processing architectures on Azure • Enhance performance and processing efficiency for large-scale simulations • Collaborate with architects and stakeholders to validate platform capabilities Requirements • Expertise in Azure Databricks and PySpark • Strong Python and SQL programming skills • Experience with Azure Data Platform and ecosystem • Knowledge of data architecture and performance optimization • Ability to support EST hours while working remotely Project Goal Demonstrate that the platform can process 100,000 simulation years in minutes instead of hours, leveraging the power of Databricks and modern data engineering practices.