Tester AI QA (Polish is Mandatory) (She/He/They) at Capco

Capco

Poland·1w ago

Mid LevelFull-timePythonRailsGCPAzureREST APIAgileCI/CDGitAI

Salary Range

$35,700 - $48,300 /year

Estimated

This salary is estimated based on similar roles. The actual salary may vary.

About the Role

We are looking for a detail-oriented QA / AI QA Tester to join our team at Capco Poland. This role focuses on testing LLM- or agent-based systems and improving the reliability and quality of enterprise AI solutions.

Key Responsibilities

→Design and maintain business test suites for the Master Agent and domain agents.
→Build evaluation datasets, including positive/negative queries and edge cases.
→Perform response quality evaluation using metrics such as accuracy, top-k recall, groundedness, hallucination rate, and refusal policy compliance.
→Conduct PII and compliance testing, including validation of masking and sensitive data handling.
→Test guardrails, including undesired output handling and prompt security testing.
→Perform performance and resilience testing, including latency and SLA compliance.
→Validate conversational UX, including conversation flow and intent recognition.
→Test integrations with various platforms and document test results.
→Support AI and domain teams in diagnosing defects and quality regression.
→Participate in knowledge quality reviews and verify compliance with KM governance rules.

Requirements

✓Experience in testing LLM-based or agent-based systems, or classical QA experience with a strong interest in transitioning to AI QA.
✓Ability to design business scenarios, test cases, and evaluation datasets.
✓Basic Python skills, including pandas, REST APIs, and simple evaluation scripts.
✓Familiarity with Copilot Studio and integration with domain agents.
✓Basic knowledge of Azure AI Search, SharePoint Online, and Document Intelligence.
✓Understanding of automated evaluation methods.
✓Fluent Polish (spoken and written) – mandatory.
✓Good command of English for documentation and collaboration.
✓Availability to work on-site, with partial remote work - 3 days per week from the office in Warsaw.