Machine Learning Engineer - LLM Evals + Observability at Gleanwork

Gleanwork

San Francisco Bay Area·4h ago

Mid LevelFull-timePythonRustMachine LearningGitGoAIML

Salary Range

$102,000 - $138,000 /year

Estimated

This salary is estimated based on similar roles. The actual salary may vary.

About the Role

Glean is seeking a Quality Engineer to join our team responsible for the measurement and quality layer that makes our AI Assistant and Agents reliably better over time. This role involves building evaluation pipelines, quality evalsets, LLM-powered judges, and observability infrastructure.

Key Responsibilities

→Design and curate evaluation datasets for reliable coverage of real assistant behavior
→Build and maintain large-scale evaluation pipelines to measure assistant quality
→Develop LLM-powered judges that score metrics like correctness and response quality
→Evaluate new models and product changes before they ship
→Build observability infrastructure for AI agents
→Close the loop between quality measurement and improvement
→Collaborate with engineers to integrate evaluation into our development process

Requirements

✓2+ years of software engineering experience with strong coding skills
✓Strong backend fundamentals in Go and Python
✓Experience with distributed data pipelines
✓Knowledge of LLM evaluation, reinforcement learning from human feedback, or NLP
✓Analytical rigor in understanding offline metrics and real user experience
✓Ability to thrive in a customer-focused, cross-functional environment
✓Strong commitment to quality in both systems and products