AI QA Engineer – Large Language Models at Meltplan

MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. While design software optimizes use and aesthetics and construction software optimizes execution and control, MeltPlan is building the missing layer - software that optimizes decisions and tradeoffs upstream, before scope is locked, procurement begins, and change orders become inevitable. MeltPlan’s long-term goal is to help teams make construction “boring” by making planning more intense: surfacing constraints and tradeoffs early, aligning stakeholders before plans are frozen, and reducing the need for late-stage redlines, rework, and change orders. MeltPlan is founded by operators who have built at scale. Kanav previously co-founded Innovaccer, a $3Bn healthtech company focused on making US healthcare more affordable and accessible. He’s now applying that systems-level thinking to construction.He’s joined by Tanmaya Kala, former Project Executive at DPR Construction, who led large commercial, healthcare, and life sciences projects. We combine deep tech scale with real construction execution. What This Role Really is : We are seeking a detail-oriented and technically strong AI QA Engineer to ensure the quality, reliability, and performance of Large Language Model (LLM)-based systems. You will be responsible for designing and executing test strategies, validating model outputs, and building evaluation frameworks to improve the accuracy and safety of AI-driven applications. What You'll Do: • Design and implement test strategies for LLM-based applications, including functional, regression, and performance testing • Evaluate LLM outputs for accuracy, consistency, bias, and hallucinations • Develop and maintain automated testing frameworks for AI/ML systems • Create test datasets, prompts, and evaluation benchmarks for model validation • Collaborate with AI/ML engineers, data scientists, and product teams to improve model performance • Perform prompt testing, prompt tuning validation, and response quality analysis • Identify edge cases, failure scenarios, and model vulnerabilities • Validate API integrations and end-to-end workflows involving LLMs • Ensure compliance with responsible AI practices, including fairness, safety, and ethical standards • Document test plans, test cases, and quality metrics You are responsible for making MeltPlan work in the real world. What We're looking for: • Bachelor’s degree in Computer Science, Engineering, or related field • 3–6 years of experience in QA/testing, preferably in AI/ML or data-driven systems • Strong understanding of software testing methodologies and QA processes • Familiarity with Large Language Models and Generative AI concepts • Experience with API testing tools (e.g., Postman) and automation frameworks • Proficiency in at least one programming language (Python preferred) • Understanding of NLP concepts such as tokenization, embeddings, and text generation • Strong analytical and problem-solving skills • Experience testing AI/ML models or data pipelines • Experience with prompt engineering and prompt testing • Familiarity with cloud platforms (AWS, GCP, or Azure) • Exposure to AI safety, bias detection, and model governance Bonus if you: • Have worked in construction or on project sites • Have startup experience • Can write code to prototype or patch solutions • Enjoy being close to the field, not just behind a desk We’re not looking for someone who waits for clean requirements.We’re looking for someone who thrives in the mess and turns it into systems. Why meltplan • Massive industry, real-world impact • High ownership from day one • Small team, zero bureaucracy • Competitive comp + meaningful equity