Loading...
Mid LevelFull-timeTypeScriptPythonRailsAI
Salary Range
$102,000 - $138,000 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for a Full Stack Developer to join Mistral AI, a pioneering company shaping the future of AI. This role involves building the evaluation and A/B testing framework, adding end-to-end observability, and running a reliable model release process.
Key Responsibilities
- →Build and maintain an LLM evaluation framework (reference tests, heuristics, model-graded checks).
- →Define and track metrics: task success, helpfulness, hallucination proxies, safety flags, latency/cost.
- →Run A/B tests for prompts, models, and system prompts, analyze results, recommend rollout or rollback.
- →Set up observability for LLM calls: structured logging, tracing, dashboards, alerts.
- →Operate the model release: canary and shadow traffic, sign-offs, SLO-based rollback criteria, regression detection.
- →Improve core behaviors: memory write/retrieve policies and evals, intent classification, follow-ups, routing, tool-call reliability.
- →Create templates and docs so other teams can author evals and ship safely.
- →Partner with Science, diagnose regressions and lead post-mortems.
Requirements
- ✓Strong TypeScript or Python skills
- ✓Production LLM experience: prompts, tool/function calling, and system prompts.
- ✓Hands-on with evals and A/B testing, you can design metrics and make rollout decisions from data.
- ✓Observability: logging, tracing, dashboards, alerting
- ✓Product mindset: form hypotheses, run experiments, interpret results, iterate.
- ✓Clear written and spoken communication, autonomous; and product-oriented.
Benefits
- ★Competitive salary and equity
- ★Health insurance
- ★Transportation allowance
- ★Sport allowance
- ★Meal vouchers
- ★Private pension plan
- ★Generous parental leave policy
- ★Visa sponsorship
