Lead Data Scientist - GenAI at Blend360

About the Role

We are looking for a GenAI Delivery Scientist to own the scientific and methodological aspects of GenAI delivery. This role involves problem framing, feasibility assessment, experimental design, evaluation strategy, metric selection, ground-truth creation, and decisioning on model and prompting approaches.

Key Responsibilities

• →Translate business needs into testable GenAI hypotheses, clear outputs, and measurable success criteria; define scope boundaries (what the system should not attempt), including risks.
• →Run feasibility assessments to choose the right approach: prompting vs RAG vs fine-tuning vs classical ML.
• →Select and develop models based on task requirements (reasoning vs extraction vs classification) working with AI Engineering to understand latency/cost, and risk profile.
• →Design prompting strategies: instruction design, few-shot sets, structured outputs, tool/agent prompts, and robustness patterns. This will be implemented as an MVP and iterate based on eval results.
• →Establish prompt iteration methodology driven by evals (not anecdotal testing): prompt versioning, ablations, and change control.
• →Define the evaluation plan for GenAI systems and agentic workflows- designing and implementing evaluation from LLM as a judge, thresholds and metric creation i.e. recall@k, precision@k. Ensure evaluation includes fairness and bias considerations where applicable. Define acceptance thresholds and release (go/no-go) gates tied to these metrics.
• →Own experimentation and model improvements: Run structured experiments (across prompts, retrievers, chunking, models).
• →Develop methods for identifying model failures such as hallucination types, retrieval misses, instruction-following errors, formatting failures etc
• →Provide recommendations for improvements grounded in evidence: what to change, expected lift, and tradeoffs.
• →Deliver an engineering-ready handoff: prompt packages and versioning approach, RAG configuration, tool schemas (if agentic), evaluation harness, datasets/ground truth, metric definitions, and go/no-go gates.
• →Required Collaboration Model
• →Act as the GenAI DS lead in project delivery: align stakeholders on success metrics, evaluation readouts, and go/no-go decisions.
• →Partner AI engineering for LLM implementation needs by providing clear specs (prompts/tool schemas), eval harnesses, and acceptance thresholds.
• →Mentor DS/analysts on GenAI evaluation methods, labelling operations, and scientific rigor.
• →With Product and Software Engineers for integrating AI capabilities into platforms and user-facing services.
• →With DevOps/Platform Engineers for environment setup, monitoring, infrastructure, and reliability.
• →With Data Engineering for designing and accessing upstream data pipelines.