AI Query Evaluation Specialist (Copilot Competitive Intelligence) at Bpcs

Who is Blueprint? We are a technology solutions firm headquartered in Bellevue, Washington, with a strong presence across the United States. Unified by a shared passion for solving complicated problems, our people are our greatest asset. We use technology as a tool to bridge the gap between strategy and execution, powered by the knowledge, skills, and the expertise of our teams, who all have unique perspectives and years of experience across multiple industries. We’re bold, smart, agile, and fun. What does Blueprint do? Blueprint helps organizations unlock value from existing assets by leveraging cutting-edge technology to create additional revenue streams and new lines of business. We connect strategy, business solutions, products, and services to transform and grow companies. Why Blueprint? At Blueprint, we believe in the power of possibility and are passionate about bringing it to life. Whether you join our bustling product division, our multifaceted services team or you want to grow your career in human resources, your ability to make an impact is amplified when you join one of our teams. You’ll focus on solving unique business problems while gaining hands-on experience with the world’s best technology. We believe in unique perspectives and build teams of people with diverse skillsets and backgrounds. At Blueprint, you’ll have the opportunity to work with multiple clients and teams, such as data science and product development, all while learning, growing, and developing new solutions. We guarantee you won’t find a better place to work and thrive than at Blueprint. We are looking for an AI Query Evaluation Specialist (Copilot Competitive Intelligence) to join us as we build cutting-edge technology solutions! This is your opportunity to be part of a team that is committed to delivering best-in-class service to our customers. In this role, you will support a high-impact initiative focused on evaluating and improving the quality of AI-powered search and assistant experiences. You will work with real user queries to build and refine evaluation datasets used to benchmark Microsoft Copilot against leading AI systems. This role sits at the intersection of language, data, and product insight, requiring both analytical rigor and strong intuition around user intent. This role requires alignment with client best practices and safety protocols. Responsibilities: • Review and analyze real user query logs to identify queries with clear intent and strong representativeness of English-speaking markets • Curate and maintain high-quality datasets used to evaluate AI systems such as Microsoft Copilot, ChatGPT, and Gemini • Annotate queries across multiple evaluation dimensions, including: • Need for web search or external information retrieval • Presence or likelihood of personally identifiable information (PII) • Requirement for domain-specific or professional expertise • Additional attributes relevant to AI response quality • Ensure annotations are consistent, structured, and aligned with evolving evaluation guidelines • Use tools such as Excel to organize, review, and summarize evaluation outputs • Identify trends in query patterns and provide feedback to improve dataset coverage and quality • Maintain a high level of attention to detail, documentation quality, and evaluation integrity Qualifications: • Strong English reading comprehension with the ability to interpret subtle differences in user intent • Demonstrated analytical thinking and logical reasoning skills • Experience working with structured data or annotation workflows • Familiarity with tools such as Microsoft Excel or similar data analysis tools • Strong user empathy and understanding of how diverse users formulate queries • Curiosity and familiarity with modern AI tools (e.g., Copilot, ChatGPT, Gemini) • High attention to detail with a track record of delivering consistent, high-quality work • Reliable, proactive, and adaptable in fast-changing environments Preferred Qualifications: • Prior experience in data annotation, content evaluation, or dataset curation for AI or search products • Experience with AI evaluation, search relevance, or linguistic analysis • Basic statistical or data analysis knowledge • Demonstrated ability to quickly learn and interpret unfamiliar domains • Fluency in English plus at least one additional language: Japanese, Korean, French, Chinese, German, or Italian Salary Range Pay ranges vary based on multiple factors including, without limitation, skill sets, education, responsibilities, experience, and geographical market. The pay range for this position reflects geographic based ranges for Washington state: $80,000 – $95,000 USD annually, with a midpoint of $87,500. Our compensation philosophy is to align offers to experience and internal equity. As such, offers are typically not made at the top of the range and are more commonly aligned closer to the midpoint. The salary/wage and job title for this opening will be based on the selected candidate’s qualifications and experience and may be outside this range. Equal Opportunity Employer Blueprint Technologies, LLC is an equal employment opportunity employer. Qualified applicants are considered without regard to race, color, age, disability, sex, gender identity or expression, orientation, veteran/military status, religion, national origin, ancestry, marital, or familial status, genetic information, citizenship, or any other status protected by law. If you need assistance or a reasonable accommodation to complete the application process, please reach out to: [Upgrade to PRO to see contact] Blueprint believes in the importance of a healthy and happy team, which is why our comprehensive benefits package includes: • Medical, dental, and vision coverage • Flexible Spending Account • 401k program • Competitive PTO offerings • Parental Leave • Opportunities for professional growth and development Location: Remote (U.S.-based)