Senior Data Engineer at Hitachidigitalservices

Our Company We’re Hitachi Digital Services, a global digital solutions and transformation business with a bold vision of our world’s potential. We’re people-centric and here to power good. Every day, we future-proof urban spaces, conserve natural resources, protect rainforests, and save lives. This is a world where innovation, technology, and deep expertise come together to take our company and customers from what’s now to what’s next. We make it happen through the power of acceleration. Imagine the sheer breadth of talent it takes to bring a better tomorrow closer to today. We don’t expect you to ‘fit’ every requirement – your life experience, character, perspective, and passion for achieving great things in the world are equally as important to us. The Team You will be part of a high-impact Data & Analytics engineering team focused on building scalable, cloud-native data platforms that power enterprise analytics and mission-critical business applications. The team collaborates closely with data architects, platform leads, and cross-functional stakeholders to design modern data lake architectures and real-time data ecosystems on AWS. The Role As a Senior Data Engineer, you will play a hands-on role in designing, building, and operating high-performance batch and streaming data platforms. You will: • Design, develop, and maintain large-scale batch and streaming pipelines using PySpark and Python. • Build real-time and near real-time streaming applications with stateful processing, windowing, and checkpointing. • Develop production-grade Python microservices for complex data transformations and business logic. • Design and manage modern data lake architectures using Apache Iceberg on AWS S3, implementing schema evolution, partitioning, compaction, and time travel. • Develop and deploy pipelines across AWS services including S3, EMR, Glue, Lambda, Athena, Redshift, and Aurora. • Optimize Spark workloads for performance, scalability, and cost efficiency. • Implement monitoring, logging, alerting, and recovery mechanisms for robust production operations. • Contribute to CI/CD pipelines, participate in architecture discussions, and uphold engineering best practices. What You’ll Bring • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline. • Strong hands-on expertise in PySpark, Spark SQL, and distributed data processing. • Advanced proficiency in Python for building scalable, production-grade data solutions and microservices. • Proven experience building and running Kafka-based streaming applications in production environments. • Deep understanding of streaming fundamentals, including stateful processing and fault tolerance. • Hands-on experience with Apache Iceberg in production data lake environments. • Solid experience with AWS data services (S3, EMR, Glue, Lambda, Redshift, Aurora). • Advanced SQL skills and strong knowledge of data modeling and modern data lake architectures. • Strong troubleshooting skills in distributed data systems with a focus on reliability and performance. About us We’re a global, team of innovators. Together, we harness engineering excellence and passion to co-create meaningful solutions to complex challenges. We turn organizations into data-driven leaders that can make a positive impact on their industries and society. If you believe that innovation can bring a better tomorrow closer to today, this is the place for you. #LI-RS2 Fostering innovation through diverse perspectives Hitachi is a global company operating across a wide range of industries and regions. One of the things that sets Hitachi apart is the diversity of our business and people, which drives our innovation and growth. We are committed to building an inclusive culture based on mutual respect and merit-based systems. We believe that when people feel valued, heard, and safe to express themselves, they do their best work. How we look after you We help take care of your today and tomorrow with industry-leading benefits, support, and services that look after your holistic health and wellbeing. We’re also champions of life balance and offer flexible arrangements that work for you (role and location dependent). We’re always looking for new ways of working that bring out our best, which leads to unexpected ideas. So here, you’ll experience a sense of belonging, and discover autonomy, freedom, and ownership as you work alongside talented people you enjoy sharing knowledge with. We’re proud to say we’re an equal opportunity employer and welcome all applicants for employment without attention to race, colour, religion, sex, sexual orientation, gender identity, national origin, veteran, age, disability status or any other protected characteristic. Should you need reasonable accommodations during the recruitment process, please let us know so that we can do our best to set you up for success.