Loading...
Senior Product Manager – DPAI Arena at Jetbrains
JetbrainsAmsterdam, Netherlands; Belgrade, Serbia; Berlin, Germany; Limassol, Cyprus; Madrid, Spain; Munich, Germany; Paphos, Cyprus; Prague, Czech Republic; Tel Aviv, Israel; Warsaw, Poland; Yerevan, Armenia·14h ago
SeniorFull-timeAIML
Salary Range
$143,650 - $194,350 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for a Product Manager to own the direction of DPAI Arena, a project aimed at defining and maintaining an open, community-driven benchmark for evaluating AI agents and IDE-embedded AI features at scale. This role will help define what “good evaluation” looks like and how we measure it.
Key Responsibilities
- →Refine our answer to the question of what makes an evaluation task a representative data point for software engineers and IDE AI features, using usage data, feedback loops, and community input.
- →Define and iterate the evaluation methodology, designing tasks, specifying metrics and success criteria, and covering real developer workflows from both product and UX perspectives.
- →Build and maintain relationships with external vendors, platform partners, outsourcing providers, and internal stakeholders to expand participation in DPAI Arena and drive both business and ecosystem development.
- →Curate and manage datasets, coordinating contributions from JetBrains teams, external partners, and outsourcing vendors (including crowdsourced workers when needed) to keep evaluation tasks high-quality, representative, and diverse.
- →Own the evaluation standard and roadmap, representing DPAI Arena internally and externally while aligning plans with the broader JetBrains AI and IDE strategy.
Requirements
- ✓Proven experience in product management (developer tools or data-driven products is a plus).
- ✓Experience working on GenAI features for developer tooling.
- ✓Strong analytical skills, with the ability to define meaningful metrics, interpret results, and turn them into product design.
- ✓Experience working with multiple stakeholders: internal teams, external partners, and vendors (including outsourcing providers).
- ✓Understanding of developer workflows, IDEs, and AI-assisted development, and the ability to translate real-world coding tasks into evaluation workloads.
- ✓Strong communication skills, both written and spoken.
- ✓Ownership mindset and proactive approach.
- ✓Engineering background (particularly SWE or QA for tech products).
