Loading...
SeniorFull-timeCI/CDAIAPI
Salary Range
$66,300 - $89,700 /year
This salary is estimated based on similar roles. The actual salary may vary.
Build the quality engine behind enterprise AI.
We’re looking for a hands-on QA leader to drive the strategy and execution of quality assurance for enterprise-grade Generative AI assistant applications.
You’ll own how AI systems are tested, validated, and scaled, spanning full-stack applications powered by LLMs and RAG workflows. This role is critical to ensuring reliable AI outputs, strong user experiences, and responsible, ethical AI behavior in production.
This position will offer flexibility for hybrid work schedules to include both in-office presence and telecommute/virtual work, to be based in Chicago, IL.
Key Responsibilities
• Define and execute QA strategy for end-to-end AI applications (frontend, backend, and AI workflows)
• Lead validation of LLM outputs for accuracy, coherence, and reliability across diverse scenarios
• Test and optimize RAG pipelines to ensure high-quality retrieval and grounded responses
• Design user-centric test scenarios (multi-turn conversations, edge cases, failure handling)
• Conduct performance, load, and latency testing across AI and application layers
• Build and scale automated testing frameworks (API, UI, integration) within CI/CD pipelines
• Establish and track quality metrics (accuracy, relevance, latency, user satisfaction)
• Ensure ethical AI standards, including bias detection and responsible output validation
• Partner cross-functionally to embed QA practices across the development lifecycle
Free to sign up. Takes 30 seconds.
About AECOM2
AECOM2 is a specialized consulting and engineering firm operating at the intersection of infrastructure development and technology innovation. Headquartered in a major metropolitan area, the company builds upon the legacy of its namesake parent organization, AECOM, to deliver cutting-edge solutions for complex projects worldwide. With a focus on forward-thinking approaches to urban development, transportation systems, and environmental sustainability, AECOM2 has established itself as a forward-looking entity in the global engineering and consulting sector. The "2" in its name suggests a second iteration or specialized branch of the original firm, positioning it as an evolution in service delivery and technical expertise. AECOM2 offers a comprehensive suite of services including advanced infrastructure planning, smart city technology integration, sustainable design consulting, and digital transformation solutions for public and private sector clients. Key features of their offerings include data-driven decision support systems, modular construction expertise, and lifecycle management strategies that optimize long-term performance. The firm primarily serves municipal governments, transportation authorities, energy utilities, and large-scale development corporations seeking innovative approaches to infrastructure challenges. Their solutions are characterized by interdisciplinary collaboration and the application of emerging technologies to traditional engineering problems. Notably, AECOM2 has positioned itself as a leader in integrating artificial intelligence and predictive analytics into traditional engineering workflows, creating a unique value proposition that combines proven methodologies with next-generation capabilities. This technological focus has earned them recognition in industry publications and positioned them as a forward-thinking alternative within the competitive engineering consulting market.
