Software Engineering & DevOps AI Rater/Evaluator at Lilt Production

Lilt Production

Saudi Arabia (remote)·3h ago

Mid LevelContractRustAWSCI/CDAI

Salary Range

$102,000 - $138,000 /year

Estimated

This salary is estimated based on similar roles. The actual salary may vary.

About the Role

We are seeking software engineering and DevOps professionals to contribute expert judgment to human-in-the-loop AI evaluation workflows. This role involves evaluating AI outputs related to software engineering, DevOps, and infrastructure topics, ensuring high-quality AI evaluation across training, benchmarking, red-teaming, and ongoing model monitoring.

Key Responsibilities

→Evaluate AI outputs related to software engineering, DevOps, and infrastructure topics
→Perform structured scoring, comparison, classification, and judgment tasks
→Assess technical correctness, completeness, security implications, and best-practice alignment
→Identify hallucinations, incorrect code, unsafe recommendations, or misleading system guidance
→Apply domain-specific engineering and DevOps guidelines consistently across tasks

Requirements

✓Deep domain expertise in software engineering, DevOps, or infrastructure
✓Strong technical judgment and ability to apply criteria consistently
✓Comfort working with structured evaluation workflows
✓Ability to explain reasoning clearly, especially in complex or high-risk technical scenarios
✓Reliability, professionalism, and respect for quality standards