QA Engineer – Agentic AI Systems

Job Title: Lead / Senior QA Engineer – Agentic AI Systems (EST Overlap Preferred) Experience 12+ VISA : Any VISA Client: Anonymous health(Remote) Role Overview We are looking for a highly skilled QA professional to build and scale a next-generation Agentic AI Quality Engineering function. This role goes beyond traditional QA—focusing on validating autonomous AI systems, designing evaluation frameworks, and ensuring high-quality outputs across multiple AI-driven products. You will play a critical role in shaping how quality is defined, measured, and improved for agentic systems that operate with minimal human intervention. Key Responsibilities Establish best practices for testing AI agents across lifecycle stages 2. Product Quality Ownership Own QA for 3 core AI products: AI Contact Center solutions AI Chat & Form-based interaction systems AI Assistants (autonomous / semi-autonomous agents) Define quality benchmarks, SLAs, and success metrics for each product Proactively identify quality gaps ahead of customer impact 3. Metrics, Observability & Evaluation Define and track performance outputs for agentic systems (accuracy, latency, resolution quality, hallucination rate, etc.) Build frameworks for: Evals & graders (LLM evaluation pipelines) Output scoring and benchmarking Continuous feedback loops Leverage tools like Langfuse for: LLM observability and tracing Prompt monitoring and performance analysis Debugging agent behavior in production Analyze: Downstream issues Production tickets Failure patterns 4. Automation & Testing Frameworks Build and scale automation across: Regression testing Smoke testing End-to-end agent workflows Develop and maintain Playwright-based automation scripts Integrate QA into CI/CD pipelines for continuous validation 5. Agentic Testing & Validation Design testing approaches for: Multi-step agent workflows Context retention and reasoning Tool usage by agents Work with orchestration frameworks like Temporal to: Validate long-running workflows Test retries, state transitions, and failure handling in agent pipelines Account for non-deterministic behavior in AI systems Invest additional effort in agentic validation, recognizing higher complexity vs traditional QA 5–10+ years in QA / Quality Engineering, with strong automation experience Hands-on experience with: Test automation tools (Playwright preferred) API and system testing Strong understanding of: AI/ML systems (LLMs, conversational AI preferred) Evaluation frameworks and benchmarking Experience with: Temporal (workflow orchestration, stateful systems testing) Langfuse (LLM observability, tracing, and evaluation) Experience in: Building QA frameworks from scratch Working with production data, logs, and issue triaging Good to Have Experience with LLM eval frameworks, prompt testing, or AI red-teaming QA evolving from reactive testing to proactive quality intelligence [Upgrade to PRO to see contact]