Loading...
Mid LevelFull-timeTypeScriptPythonGoRailsAPIAI
About the Role
We are looking for an AI Engineer to join our product team at Mistral AI. This role focuses on building evaluation frameworks, implementing A/B testing systems, and ensuring reliable model releases to improve AI quality, latency, safety, and reliability.
Key Responsibilities
- →Build and maintain an LLM evaluation framework including reference tests and model-graded checks
- →Define and track metrics such as task success, helpfulness, hallucination proxies, and safety flags
- →Run A/B tests for prompts, models, and system prompts, analyzing results to recommend rollout or rollback decisions
- →Set up observability for LLM calls with structured logging, tracing, dashboards, and alerts
- →Operate the model release process including canary traffic, sign-offs, and SLO-based rollback criteria
- →Improve core AI behaviors such as memory policies, intent classification, and tool-call reliability
- →Create templates and documentation to help other teams author evaluations and ship safely
- →Partner with Science team to diagnose regressions and lead post-mortems
Requirements
- ✓Strong TypeScript or Python programming skills
- ✓Production LLM experience with prompts, tool/function calling, and system prompts
- ✓Hands-on experience with evaluations and A/B testing, including designing metrics and making rollout decisions
- ✓Experience with observability tools including logging, tracing, dashboards, and alerting
- ✓Product mindset with ability to form hypotheses, run experiments, interpret results, and iterate
- ✓Clear written and spoken communication skills with autonomy and product orientation
- ✓Experience with safety systems such as moderation, PII handling, and guardrails is ideal
- ✓Experience with release operations including canary/shadowing and automated rollbacks is ideal
Benefits
- ★Competitive salary and equity
- ★Health insurance
- ★Transportation allowance
- ★Sport allowance
- ★Meal vouchers
- ★Private pension plan
- ★Generous parental leave policy
- ★Visa sponsorship
