Jobs → Data & Analytics → Lead Data Engineer

About the Role We are looking for a Lead Data Engineer to join our team. This role focuses on architecting reliable data sourcing systems, building intelligent extraction frameworks, and leveraging AI tools to solve complex data challenges at scale. Key Responsibilities • Design end-to-end data pipelines covering extraction, normalization, deduplication, validation, and loading • Build scalable web scraping and extraction frameworks to handle anti-bot protections, JavaScript-heavy websites, schema drift, and proxy/IP rotation • Develop agentic extraction workflows using rule-based logic, LLM escalation, structured outputs, retries, and human-review mechanisms • Build production-ready LLM systems with prompt versioning, evaluation datasets, precision/recall tracking, observability, and drift detection • Optimize pipeline performance, infrastructure costs, concurrency, and large-scale data processing • Build reusable SKILL.md specifications, agent workflows, and orchestration systems using Airflow or equivalent tools • Use AI-powered development tools such as Claude Code, Cursor, or equivalent to build and ship production systems Requirements • 6–10 years of experience in production data engineering, ETL, or extraction systems • Strong hands-on expertise in Python and Advanced SQL • Deep experience in web scraping, large-scale extraction, anti-bot handling, proxies, and JavaScript rendering • Proven experience integrating LLMs into production extraction pipelines • Hands-on experience with agentic workflows, tool calling, structured outputs, and evaluation frameworks • Strong understanding of Airflow or equivalent orchestration tools • Experience with LLM observability, tracing, prompt versioning, evaluation, and drift detection • Practical experience shipping production solutions using Claude Code, Cursor, or similar AI coding tools • At least 1 year of hands-on experience with Snowflake • Formal people-management experience with at least 1 direct report