Senior AI Technical Product Manager - R01563914 at Brillio 2

Senior Product Manager Primary Skills • Prior hands‑on technical or data science experience • Candidates who have moved from technical or data roles into product, innovation, or applied AI will be particularly strong fit • Ability to bridge technical depth with product and business thinking Specialization • Data Science Advanced: Data Specialist Job requirements • Location: PA • Work type: Remote opportunity with occasional travel to client location • • About the Role This is a Technical Product Manager role. You’ll lead a team of engineers, researchers, and data scientists, and you need to be deep enough in the technology to QA AI agent outputs, evaluate RAG pipeline architectures, challenge prompt engineering decisions, review system designs, and guide the team’s technology roadmap. You’re not writing production code, but you’re reading it, understanding it, and making informed calls about whether the technical approach is sound. • At the same time, you’re the person who builds relationships with internal business teams, understands their pain points, and translates that into the right priorities for a technical team. The difference between this role and a traditional PM is that you’re expected to be in the technical details reviewing architecture decisions, sitting in code reviews, and understanding the tradeoffs between different model choices, retrieval strategies, and deployment approaches. • You own three things: what the team works on (prioritization), how well it’s done (quality), and whether it matters (stakeholder alignment). You lead planning and serve as the quality gate every piece of work passes through your technical and product review before it ships or gets shared with stakeholders. As the team’s solutions scale, this role will also evolve into LLM cost optimization as a budget line, navigating AI governance and responsible AI practices, and guiding the technology roadmap as the AI landscape shifts. • Responsibilities Product & Stakeholder Management • Set priorities and decide what the team works on across both AI Enablement and AI Experiments tracks • Serve as the primary liaison with internal business teams understand their workflows, gather requirements, and translate pain points into actionable technical work • Collaborate with product teams to ensure exploration work aligns with the broader product direction • Lead the team’s operating rhythm stand-ups, demos, planning sessions, and progress readouts to leadership and stakeholders • Make resource allocation decisions across workstreams move people where they’re needed most based on shifting priorities • Decide when to kill work that isn’t delivering results you’re comfortable shutting down experiments and reprioritizing without hesitation • Communicate progress clearly to leadership and cross-functional partners what shipped, what we learned, what’s next, and what needs their attention • Identify new internal processes where AI could meaningfully reduce effort, and build the business case to expand the team’s scope Technical Leadership • Guide the team’s technology roadmap make informed decisions about model selection, infrastructure choices, build-vs-buy tradeoffs, and when to adopt new tools or frameworks. You stay current on how the AI landscape is evolving and what that means for the team’s technical strategy. • QA all AI outputs before they move forward review agent responses for quality, accuracy, and edge case handling. You need to understand why an agent produced a given output and whether the underlying prompt, retrieval, or orchestration logic is sound. • Review and guide architecture decisions evaluate RAG pipeline designs, agent orchestration patterns, data flow architectures, and integration approaches. You’re the person who asks “why did we choose this embedding model?” or “what happens when the retrieval layer returns irrelevant context?” and expects a rigorous answer. • Evaluate experiment designs for technical rigor when the team tests a new model or prompting approach, you review the evaluation methodology, success criteria, and whether the comparison is fair before the experiment runs, not after. • Participate in technical design reviews and code reviews not to approve every line of code, but to stay close enough to the implementation that you can catch architectural issues, scalability concerns, and technical debt before they compound. • Manage and optimize AI infrastructure spend track LLM costs, token usage patterns, and vendor contracts. You understand the cost implications of different model choices, caching strategies, and prompt designs at a technical level, not just a budget level. • Define and evolve AI governance and compliance practices as solutions handle sensitive data and higher-stakes decisions, you establish the guardrails for responsible AI use, data handling, and decision explainability. • Requirements Technical Depth (Non-Negotiable) • You’ve built things before whether earlier in your career as an engineer, data scientist, or ML practitioner, you have a hands-on technical background. You didn’t start in product management; you moved into it from a technical role. • Strong understanding of LLM-based systems you know how RAG pipelines work (retrieval, embedding, re-ranking, context window management), how prompt engineering affects output quality, and how agent orchestration patterns handle multi-step workflows. • Comfortable with Python you can read Python code, understand what it does, review pull requests for logic and architecture (not just style), and write quick scripts when needed to test an idea or validate data. • Familiar with Azure cloud infrastructure you understand how AI workloads are deployed, monitored, and scaled in Azure (or equivalent cloud platforms). You can have an informed conversation about compute costs, model serving, and infrastructure architecture. • You understand the cost structure of LLM-based systems token economics, model pricing tiers, the cost implications of different prompt lengths, caching strategies, and model routing approaches. You can optimize AI spend at a technical level, not just a budget level. • You can evaluate AI outputs critically you know what a hallucination looks like, you understand why an agent might produce inconsistent results, and you can diagnose whether the issue is in the prompt, the retrieval layer, the model choice, or the data. • Product & Leadership • 8+ years of experience in technical product management, engineering management, or a similar role where you’ve led technical teams building AI/ML or data-driven products • Experience working with internal stakeholders who aren’t technical you can translate between what a business team needs and what an engineering team should build • Experience running lean, fast-moving teams with minimal process overhead. You prefer shipping over planning. • Strong quality instinct you notice when an AI output isn’t quite right, when an experiment design has gaps, or when a prototype is missing a critical edge case • Clear communicator who can present to senior leadership without hiding behind jargon or overcomplicating things • Comfort making prioritization calls with incomplete information and adjusting course as you learn more • Experience managing budgets or vendor relationships for AI/ML infrastructure, cloud services, or technical platforms • Experience with agile methodologies (Scrum, Kanban, or similar) and the judgment to adapt the process to what the team needs • Nice to Have • Previous role as a software engineer, ML engineer, or data scientist bonus if you’ve worked on NLP, search, or content generation systems • Hands-on experience building or maintaining RAG pipelines, AI agents, or LLM-based applications • Experience with AI evaluation frameworks automated scoring, human evaluation protocols, A/B testing for AI systems • Experience with certification, assessment, or education technology • Exposure to AI governance frameworks, responsible AI practices, or compliance requirements for AI-driven decision systems • Experience scaling a team or function from early-stage to a broader organizational capability • Familiarity with MLOps practices model versioning, deployment pipelines, monitoring, and cost tracking