Loading...
Mid LevelFull-timePythonAWSDockerPostgreSQLAIML
Salary Range
$102,000 - $138,000 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for an AI Engineer (Evaluation Systems) to join GoFasti, a Talent-as-a-Service company that connects world-class developers and designers from Latin America with global companies. This role involves designing and implementing evaluation systems for AI and LLM-based systems.
Key Responsibilities
- →Design a structured, configurable evaluation engine
- →Combine deterministic checks with LLM-as-judge verdicts
- →Build calibration workflows using expert-labeled examples
- →Measure precision and recall properly (not raw agreement)
- →Handle delayed outcomes and low-confidence review flows
- →Store structured verdicts that power dashboards and analytics
- →Monitor drift and trigger recalibration when alignment drops
- →Specialized judges for specific failure patterns
- →Content-level evaluation (with careful PII handling)
- →Error analysis pipelines that drive workflow redesign
- →A/B testing for judge versions
- →Eval-driven fine-tuning data curation
Requirements
- ✓4+ years total backend / ML engineering experience
- ✓2+ years building production AI/LLM systems
- ✓Experience with Python, Docker and PostgreSQL
- ✓Experience with AWS, OpenAI, Anthropic, and other LLM APIs
- ✓Proven experience building LLM-based systems in production environments where output quality and reliability were critical
- ✓Experience developing evaluation, QA, or scoring pipelines to assess model performance
- ✓Strong understanding of precision/recall trade-offs and working with real-world data
- ✓Ability to design and implement systems that produce reliable, structured outputs from LLMs
- ✓Strong Python programming skills, including experience with asynchronous programming
- ✓Experience deploying and managing services on AWS
- ✓It's a Plus
- ✓Human-in-the-loop workflow design
- ✓OpenTelemetry familiarity
Free to sign up. Takes 30 seconds.
