Senior Data Engineer (Fabric and Databricks) at Orioninnovation

Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education. Job Overview: We are seeking an experienced Senior Data Engineer to build and optimize scalable data platforms using Microsoft Fabric, Databricks (and/or Snowflake). The role focuses on designing reliable data pipelines, lakehouse and warehouse models, and semantic layers that enable enterprise analytics, BI, and AI/Gen AI use cases. You will work closely with analytics, BI, and data science teams to deliver high-quality, performant, and governed data solutions, while driving best practices in data engineering, optimization, and platform design. Key Responsibilities • Design, build, and maintain end-to-end data solutions on Microsoft Fabric, Databricks, including Pipelines, Notebooks, Lakehouse, Data Warehouse, and Semantic Models. • Implement scalable data ingestion, transformation, and loading (ETL/ELT) using Fabric Pipelines and PySpark. • Develop robust data models and schemas optimized for analytics, reporting, and AI-driven consumption. • Create and maintain semantic models to support Power BI and enterprise BI solutions. • Engineer high-performance data solutions that meet requirements for throughput, scalability, quality, and security. • Author efficient PySpark and SQL code for large-scale data transformation, data quality management, and business rule processing. • Build reusable framework components for metadata-driven pipelines and automation. • Optimize Lakehouse and Data Warehouse performance, including partitioning, indexing, Delta optimization, and compute tuning. • Develop and maintain stored procedures and advanced SQL logic for operational workloads. • Design and prepare feature-ready datasets for AI and GenAI applications. • Collaborate with data scientists and ML engineers to productionize AI pipelines. • Implement data governance and metadata practices required for responsible AI. • Leverage Fabric, Databricks capabilities to orchestrate and monitor AI-related data workflows. • Apply data governance, privacy, and security standards across all engineered assets. • Implement monitoring, alerting, and observability best practices for pipelines and compute workloads. • Drive data quality initiatives, including validation frameworks, profiling, and anomaly detection. • Partner with analytics, BI, data science, and product teams to understand requirements and translate them into technical solutions. • Mentor junior engineers and contribute to engineering standards and patterns. • Participate in architecture reviews, technical design sessions, and roadmap planning. • Develop and deliver dashboards and reports using Power BI and Tableau. Required Skills: • Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or equivalent experience. • 7+ years of professional experience in data engineering or a related discipline. • Mandatory hands-on expertise with Microsoft Fabric, Databricks or Snowflake including: • Pipelines • Notebooks • Lakehouse • Data Warehouse • Semantic Models • Advanced proficiency in PySpark for distributed data processing. • Strong command of SQL for analytical and operational workloads. • Experience developing and optimizing stored procedures and complex SQL transformations. • Understanding of GenAI architectures, vectorization, embedding pipelines, and data preparation for LLM use cases. • Strong knowledge of data modeling, ETL/ELT patterns, and modern data lakehouse principles. • Experience designing and optimizing large-scale data pipelines in cloud environments (Azure preferred). • Excellent problem-solving skills with the ability to analyze complex data workflows. • Proficiency in creating interactive dashboards and reports using Power BI and Tableau. Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law. Candidate Privacy Policy Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains: • What information we collect during our application and recruitment process and why we collect it; • How we handle that information; and • How to access and update that information. Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.