Staff AI Linguist at LinkedIn3

This role will be based in Mountain View, CA. At LinkedIn, our approach to flexible work is centered on trust and optimized for culture, connection, clarity, and the evolving needs of our business. The work location of this role is hybrid, meaning it will be performed both from home and from a LinkedIn office on select days, as determined by the business needs of the team. HALO (Human Judgment, Annotation, Localization, and Operations) is a horizontal team within Core AI that partners across the company to enable high-quality human judgment for AI development. We partner closely with cross-functional stakeholders and internal teams to define quality goals, design evaluation and data pipelines, and scale repeatable measurement systems. Our work spans multiple initiatives at once, supported by shared standards, platforms, and best practices that help teams move faster without compromising quality.   Key Responsibilities • Lead cross-functional alignment with Engineering, Product, Data Science, domain SMEs, Trust/Legal, TPM, and vendor operations on evaluation strategy, quality goals, tradeoffs, and delivery across multiple initiatives • Define and evolve evaluation frameworks for complex model and agent behaviors, including rubrics, rating scales, defect taxonomies, escalation criteria, and market-specific guidance for ambiguous, multi-step, and high-impact use cases • Own end-to-end evaluation systems, including metrics, scorecards, regression sets, monitoring plans, scenario suites, and success criteria, and ensure outputs are repeatable, decision-useful, and adopted by partner teams • Design and operationalize annotation and evaluation pipelines across internal and vendor platforms, including task design, QA gates, adjudication approaches, workflow maintenance, and documentation • Drive development of human, synthetic, and adversarial datasets to improve evaluation coverage, identify blind spots, and support model iteration, LLM-as-a-judge systems, and reward model development • Lead calibration strategy and disagreement analysis across human and model judgments; identify drift, root causes, and reliability issues, and translate findings into guideline updates, new edge cases, retraining opportunities, and product quality improvements • Set and uphold quality standards for vendor and internal workforces, including onboarding, guideline training, audit design, escalation handling, and cost-quality tradeoff decisions across medium-to-large programs • Lead error analysis and evaluation experiments; synthesize findings into clear recommendations and influence roadmap, launch readiness, and quality investments • Define requirements for human judgment and evaluation tooling, and partner with Engineering on design, testing, rollout, and adoption • Create reusable standards and best practices that scale across teams, and enable partners on methodology, score interpretation, and appropriate use of evaluation outputs • Mentor junior team members on evaluation design, annotation quality, analysis methods, and operational excellence • Demonstrate learning agility in a rapidly evolving field by incorporating new tools, methods, and research into evaluation strategy and workflows • Apply native-speaker linguistic and cultural expertise in French (France), German (Germany), Spanish (Spain), Portuguese (Brazil), or other i18n market(s) to define market-appropriate quality standards and improve consistency across locales