Software Engineer — AI Evaluation & Automation
Ganesh shiva kumarUnited StatesMIDPosted 5h ago
PythonJavaJavaScriptAI coding toolsagentic harnessesGitDockerAPIsCI/CD pipelinesLLM-as-judgeagent-as-judgeBazel
About the Role
We are looking for a Software Engineer to build, scale, and validate evaluation harnesses that measure the performance of AI-powered software development tools. This role focuses on ensuring reproducibility and calibrating benchmarks against real-world engineering standards.
Key Responsibilities
• Build evaluation harnesses using real engineering artifacts
• Ensure reproducibility by architecting versioned evaluation pipelines
• Calibrate and validate LLM-as-judge and agent-as-judge evaluations
• Analyze performance metrics across repeated benchmark runs
• Collaborate with engineering and data teams to optimize tooling
Requirements
• Strong software engineering background with experience in Python, Java, or JavaScript
• Hands-on experience with AI coding tools and agentic harnesses
• Proven evaluation and benchmarking experience
• Solid working knowledge of Git and Docker
• Experience with APIs, CI/CD pipelines, and modern developer workflows
Preferred Skills
• Experience with LLM-as-judge and agent-as-judge calibration
• Familiarity with build-system-aware test selection
• Experience managing versioned evaluation datasets