[Evergreen] Data Engineer at Opswat

OPSWAT, a global leader in IT, OT, and ICS critical infrastructure cybersecurity, delivers an end-to-end platform that gives public and private sector organizations and enterprises the critical advantage needed to protect their complex networks, secure their devices, and ensure compliance. Over the last 20 years our commitment to innovative technology has earned the trust of more than 1,700 organizations, governments, and institutions globally, solidifying our role in protecting the world’s critical infrastructure and securing our way of life. The Position The Data Engineering team is a vital component of OPSWAT's technology division, responsible for building and maintaining the foundational data infrastructure that supports the entire organization. The team's mission is to create a scalable, reliable, and secure data platform on Microsoft Azure, enabling data-driven decision-making across all business units. This involves designing and managing data warehouses and data lakes, developing robust data integration pipelines, and ensuring the quality and accessibility of data assets. A key strategic initiative for the team is the development and implementation of a Master Data Management (MDM) layer, which will play a critical role in enhancing data consistency and accuracy across various systems and applications. What You Will be Doing • Design and Maintenance of Azure Data Warehouse/Data Lake: • Ensuring the performance, reliability, and security of the data platform, requiring proactive monitoring and optimization. • Evaluate and implementing strategies to optimize the data warehouse and data lake for cost-effectiveness and efficiency, aligning with best practices for Azure data management • Development of ETL/ELT pipelines on Azure, utilizing tools such as Azure Data Factory and Azure Databricks, to ingest and transform data from a variety of sources. • Implementing data quality rules and processes to ensure the accuracy and reliability of the data within the platform • Optimize data storage & computing resources for cost efficiency • Proactively identify pipeline bottlenecks and performance issues • Development and Implementation of Master Data Management (MDM) Layer: A key responsibility is to research and recommend suitable MDM tools and technologies available within the Azure ecosystem. Our goal is to establish a "single source of truth" for critical data entities, ensuring consistency and quality across all connected systems. • Collaboration and Communication: The individual will work closely with product teams, sales and revenue operations, customer experience, and other stakeholders across the organization to understand their data requirements and deliver effective solutions that meet their needs. This requires the ability to explain complex technical concepts in a layman terms to non-technical audiences effectively. • Risk management: ensure data warehouse availability & resiliency (during service outage, network issues, unexpected high loads, external attacks, disaster recovery, etc.) • Define effective and measurable KPIs for the team and individual team members • Team resource management & Key Man Dependency mitigations • Strategic planning (at least 3 quarters in advance on team roadmap and vision) • Study new technologies & industry trends and propose appropriate solutions for company’s need What We Need from You • Bachelor's degree in Computer Science, Data Engineering, or a related field. • Minimum of 3 years of proven experience in data warehouse/data lake design, implementation. This experience should include hands-on work with core Azure data services such as Azure Data Lake, Azure Synapse Analytics, and Azure SQL Database. • Extensive experience with ETL/ELT tools within the Azure ecosystem, including Azure Data Factory and Azure Databricks. • A strong understanding of database architecture principles, encompassing both relational databases (e.g., Azure SQL Database, Azure Synapse Analytics) and the concepts of NoSQL databases for handling unstructured data. • Experience in integrating data from multiple SaaS applications, including Salesforce, Hubspot, Netsuite, HRIS (e.g., Workday, ADP), and ZoomInfo. • Experience in researching and implementing Master Data Management (MDM) solutions. • Strong problem-solving, analytical, and troubleshooting skills are essential for this role. It Would be Nice if You Had • Good written and verbal communication skills, including the ability to collaborate effectively with cross-functional teams, are required. • Experience with real-time data streaming technologies such as Azure Event Hubs or Kafka. • Experience with DevOps practices and tools for automating data pipelines, such as Azure DevOps and CI/CD pipelines. • Experience working in a Scrum Agile team. OPSWAT is an equal opportunity employer. We celebrate diversity and are committed to providing an environment where equal employment opportunities are extended to all employees and applicants, free of discrimination and harassment of any type. All employment decisions are based on individual qualifications, job requirements, and business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws. Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.