AI Engineer at Overtone

YOUR ROLE As a founding AI Engineer at Overtone, you will build and own the AI systems that power our core product experience—from conversational intelligence to matchmaking insight generation. You’ll design prompting, evaluation, and model infrastructure that makes our AI reliable, trustworthy, and continuously improving. This is a hands-on, early-stage role: you’ll run experiments, ship infrastructure, and build the tools that give the team real visibility into model behavior and quality. What You’ll Do - LLM systems & prompt architecture: Design and maintain the prompts, structured outputs, and orchestration systems that power Overtone’s AI-driven features. - Evaluation frameworks: Build evaluation infrastructure to measure AI quality at scale using tools such as Langfuse, TensorZero, and Promptfoo. You’ll work with our team to make sure LLM judges are aligned. - Model experimentation: Run structured experiments across models, prompts, and configurations to optimize quality, cost, and latency. - AI observability & tooling: Build internal tools—dashboards, admin interfaces, debugging workflows—that give the team visibility into system behavior and model performance. - Matching & ML systems: Apply data science and lightweight modeling to improve match quality, identifying when traditional ML models or ranking systems outperform prompt-based approaches. - Model development & fine-tuning: Identify opportunities to fine-tune open-source models using techniques such as SFT, DPO, and RLHF for cost and latency improvements. - Memory & personalization systems: Design and maintain vector databases and memory architectures that enable personalization and context-aware experiences. - Responsible AI systems: Ensure systems are fair, unbiased, and auditable, building evaluation pipelines and human-in-the-loop processes where appropriate. - Cross-functional collaboration: Work closely with product, engineering, and research to bring AI capabilities into real user-facing features with scalability and reliability in mind. Who You Are You likely have: - 5+ years of engineering experience, with meaningful time spent building LLM-powered products in production - Strong prompt engineering and evaluation instincts—you think in terms of evals, not vibes - Experience building evaluation pipelines for LLM systems - Familiarity with vector databases and retrieval-augmented architectures - Experience with fine-tuning techniques such as SFT, DPO, or RLHF - Strong understanding of when ML models outperform prompt-based systems - Experience working in early-stage environments with high ownership - Experience with Python, SQLAlchemy, FastAPI, PostgreSQL Nice to have: - Experience with recommender systems or ranking systems - Experience with MLOps and model lifecycle infrastructure - Experience building AI evaluation frameworks or model observability systems - Experience building internal developer tooling (React/TypeScript + Python) COMPENSATION & BENEFITS Salary of $180k to $230k. This position will receive meaningful stock options with considerable upside potential. We provide standard benefits such as health insurance, dental insurance, etc. OUR CORE QUALITIES We look for these qualities in everyone we hire at Overtone. Curiosity: You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges. Craft: You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made. Collaboration: You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes. Care: You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust—with users, teammates, and the work itself. ABOUT OVERTONE Overtone, led by the founder and former CEO of Hinge, is building a new kind of dating service – one that cuts through the noise and helps daters find someone who truly resonates. We’re building a team who are excited to work together in person in our NYC office.