Jobs → Engineering → GenAI Evaluation Engineer

About the Role We are seeking a GenAI Evaluation Engineer to join our team. This role involves evaluating AI/ML systems, particularly in NLP and applied data science. Key Responsibilities • Develop and implement evaluation frameworks for AI/ML systems • Conduct hands-on RAG, LLM, and agent development • Apply statistical analysis techniques for evaluation • Perform LLM-as-a-judge evaluations and calibration • Implement hallucination detection and evidence traceability • Integrate CI/CD for regression monitoring • Develop expert annotation and evaluation datasets Requirements • Expertise in AI/ML evaluation, NLP, and applied data science • Experience with Python evaluation frameworks and test harnesses • Knowledge of information retrieval metrics: Recall@K, MRR, nDCG • Understanding of precision, recall, and completeness benchmarking • Familiarity with statistical analysis: confidence intervals, Cohen’s Kappa, Krippendorff’s Alpha • Experience with FDA/EMA Guidance, CTD Dossiers, Clinical Study Reports (CSRs), and GxP environments preferred