Loading...
LeadFull-timePythonCI/CDGitGoAI
Salary Range
$153,000 - $207,000 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for an Evals Engineer to join P-1 AI, a company building an engineering AGI. In this role, you will be responsible for developing and implementing evaluation systems to ensure our AI engineer, Archie, is learning and retaining the skills needed for engineering work.
Key Responsibilities
- βImplement the system for organizing, transforming, running, grading, and reporting on eval benchmarks
- βDesign and execute the process for developing and QA-ing our evals, incorporating contributions from engineering team, industrial partners, and subject-matter experts
- βEnsure evals run effectively within our CI/CD system, continuously benchmarking our evolving AI platform
- βCreate methods for detecting and testing for common quality challenges of AI
- βBe a technical leader in the consistent implementation and organization of automated tests
Requirements
- βExperience in constructing comprehensive test suites for software and/or AI systems
- βExperience designing metrics to evaluate systems and visualize their performance
- βExperience in developing, managing, and running evals against LLM-based systems is a strong plus
- βGood communication skills with a variety of stakeholders
- βProficiency in Python programming, complex modules, and modern software development tools
- βAbility to thrive in a fast-paced, dynamic startup environment
Benefits
- β Competitive salary and significant equity component
- β Healthcare, dental, and vision insurance
- β 401k with employer matching
- β Unlimited PTO
Free to sign up. Takes 30 seconds.
