Loading...
SeniorFull-timePythonMachine LearningData ScienceAI
Salary Range
$93,925 - $127,075 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for a Manager – AI Performance & Analytics to lead a team responsible for analyzing conversational AI performance and driving continuous improvements to Netomi’s AI platform. This role sits at the intersection of data analytics, conversational AI, and LLM evaluation.
Key Responsibilities
- →Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities.
- →Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction.
- →Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau.
- →Design frameworks to evaluate the response quality of LLM responses across dimensions such as factuality and groundedness, relevance to user intent, instruction adherence, tone and customer experience, and safety and compliance.
- →Build scalable processes for LLM-based automated evaluation and human review workflows.
- →Identify systematic failure patterns in AI responses and recommend improvements.
- →Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities.
- →Design and analyze experiments to measure the impact of model improvements.
- →Generate insights that inform model training data, prompts, and product features.
- →Identify trends in customer interactions to uncover new AI automation opportunities, gaps in intent coverage, and common customer pain points.
- →Translate analytics insights into actionable recommendations for product and AI teams.
- →Lead and mentor a team of data analysts focused on AI performance and conversational analytics.
- →Establish best practices for analytics, experimentation, and AI evaluation.
- →Collaborate closely with product, AI research, engineering, and customer success teams.
Requirements
- ✓7–10 years of experience in data analytics, data science, or AI analytics.
- ✓Experience managing or mentoring analytics teams.
- ✓Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.
- ✓Strong SQL and data analysis skills.
- ✓Experience with BI tools such as Tableau or similar platforms.
- ✓Experience using Python for data analysis or experimentation.
- ✓Familiarity with machine learning or large language models (LLMs).
- ✓Experience evaluating AI or machine learning model performance.
- ✓Experience working with conversational AI platforms or chatbots.
- ✓Experience with experimentation frameworks and A/B testing.
Free to sign up. Takes 30 seconds.
