LLM Quality Engineer at AECOM2

In AECOM’s AI Engineering team your code will directly shape the physical world around us. We build AI-driven technology that revolutionises how infrastructure and buildings are designed and engineered; reducing waste, cutting CO₂, and making the built environment more efficient and sustainable. This is where software has measurable, real-world impact.  With our AI Engineering team we’ve created a unique setup: a lean, highly technical team with the speed and ownership of a start up, backed by the scale, resources, and domain expertise of one of the world’s leading engineering firms.   There has never been a better time to be at AECOM. We are leading the industry’s AI transformation, and with our people and technology we deliver excellence and innovate with impact.  We invite you to bring your bold ideas and big dreams to solve the world’s most complex challenges. We're one global team driven by our common purpose to deliver a better world. Join us.  What You’ll Do  As part of the Assurance & Insight team, you will focus on ensuring the quality and reliability of our AI-driven systems. This role sits at the intersection of testing, AI evaluation, and operational quality.   You will help build the foundations for how we test, benchmark, and monitor both software systems and LLM-driven outputs ensuring that what we deploy performs as expected in real-world scenarios.  You will take ownership of quality across our AI systems, from automated testing to LLM evaluation and production monitoring. Your work will help reduce regressions, improve reliability, and free up engineering time to focus on building.  We're not looking for years of experience. We're looking for someone detail-oriented who wants to own a domain, can write clean Python, and is genuinely curious about how AI systems behave. It's a good fit for a recent graduate who wants end-to-end ownership early in their career, is comfortable working close to AI systems, and cares about making quality measurable rather than subjective.