GenAI QA Engineer
Shikha Singh K.RemoteMIDPosted 2h ago
GenAILLM Application TestingRAG architecturepipeline validationLLM evaluation frameworksRagasTruLensDeepEvalLLM evaluation metricsFaithfulnessAnswer RelevanceContext Precision/RecallHallucination RatePrompt InjectionJailbreakAI Safety TestingPythonTypeScriptAPI TestingPostmanBrunocURLAutomationPyTestPlaywrightCypressGCP servicesVertex AIBigQueryCloud StorageCloud Logging
About the Role
We are looking for a GenAI QA Engineer with expertise in RAG and GCP to join our team. This role focuses on testing GenAI and LLM applications, ensuring quality and reliability.
Key Responsibilities
• Design and implement test cases for GenAI and LLM applications
• Validate RAG architecture and pipelines
• Evaluate LLM performance using frameworks like Ragas, TruLens, or DeepEval
• Test for Prompt Injection, Jailbreak, and AI Safety
• Automate testing processes using Python or TypeScript
• Perform API testing using tools like Postman, Bruno, or cURL
• Utilize GCP services such as Vertex AI, BigQuery, Cloud Storage, and Cloud Logging
Requirements
• 4–8 years of experience in QA Automation or Software Testing
• Hands-on experience in GenAI / LLM Application Testing
• Strong understanding of RAG architecture and pipeline validation
• Experience with LLM evaluation frameworks: Ragas, TruLens or DeepEval
• Knowledge of LLM evaluation metrics – Faithfulness, Answer Relevance, Context Precision/Recall and Hallucination Rate
• Experience in Prompt Injection, Jailbreak and AI Safety Testing
• Strong Python or TypeScript programming skills
• API Testing using Postman, Bruno or cURL
• Automation experience with PyTest, Playwright or Cypress
• Working knowledge of GCP services – Vertex AI, BigQuery, Cloud Storage and Cloud Logging