Jobs → Engineering → Software Engineer — AI Evaluation & Automation

Software Engineer — AI Evaluation & Automation

Ganesh shiva kumarUnited StatesMIDPosted 5h ago
PythonJavaJavaScriptAI coding toolsagentic harnessesGitDockerAPIsCI/CD pipelinesLLM-as-judgeagent-as-judgeBazel
About the Role We are looking for a Software Engineer to build, scale, and validate evaluation harnesses that measure the performance of AI-powered software development tools. This role focuses on ensuring reproducibility and calibrating benchmarks against real-world engineering standards. Key Responsibilities • Build evaluation harnesses using real engineering artifacts • Ensure reproducibility by architecting versioned evaluation pipelines • Calibrate and validate LLM-as-judge and agent-as-judge evaluations • Analyze performance metrics across repeated benchmark runs • Collaborate with engineering and data teams to optimize tooling Requirements • Strong software engineering background with experience in Python, Java, or JavaScript • Hands-on experience with AI coding tools and agentic harnesses • Proven evaluation and benchmarking experience • Solid working knowledge of Git and Docker • Experience with APIs, CI/CD pipelines, and modern developer workflows Preferred Skills • Experience with LLM-as-judge and agent-as-judge calibration • Familiarity with build-system-aware test selection • Experience managing versioned evaluation datasets

Similar projects

AI/ML & Constraint Programming Engineer
Soma Naga Raju · CP-SAT, Google OR-Tools, Constraint Programming
Azure DevOps Engineer
Manik B · Azure, AKS, Kubernetes
SRE / DevOps Engineer
Sapna Daniel · Site Reliability Engineering (SRE), DevOps, Kubernetes
Early · 0 applied1 of 5,611 projects

Apply with AI cover letter

AI writes a personalized application in 19 seconds. Just upload your resume.

No credit card · First application free
© 2026 Freelanly · Terms · Privacy