Loading...
LeadFull-timePythonGolangRustVueMongoDBElasticsearchREST APIAIAPI
Salary Range
$153,000 - $207,000 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are seeking a Principal Engineer to lead the development of our high-scale retrieval systems and Retrieval-Augmented Generation (RAG) construction. This role is critical in bridging the gap between massive, licensed datasets and real-time generative inference.
Key Responsibilities
- →Design and implement the end-to-end RAG construction pipeline, ensuring high-performance ingestion and transformation of diverse datasets in near real-time.
- →Develop and optimize hybrid retrieval strategies that combine the precision of full-text search with the contextual depth of semantic (vector) search.
- →Own the 'document-to-chunk' lifecycle, implementing advanced strategies for chunking, metadata enrichment, and quality filtering.
- →Architect systems to handle jobs across millions of documents while optimizing indices for sub-second latency and high-throughput serving.
- →Recommend and implement optimizations for GPU/CPU performance, concurrency, and memory management.
- →Act as a technical expert, guiding the team in software design and providing diagnostic skills for complex distributed system issues.
Requirements
- ✓15+ years in Engineering with at least 8+ years architecting large-scale distributed systems.
- ✓Proven track record in Python and Golang or Rust for building commercial-grade software.
- ✓Expert-level mastery in architecting and scaling high-throughput indexing pipelines using ElasticSearch, OpenSearch, or distributed vector databases.
- ✓Deep hands-on experience optimizing large-scale search architectures for millions of documents.
- ✓Expertise in both SQL and NoSQL databases (e.g., MongoDB, Clickhouse, Postgres) and big data processing tools.
- ✓Excellent knowledge of algorithms, data structures, graph theory, and modern distributed application principles.
- ✓MS in Computer Science or Engineering
Free to sign up. Takes 30 seconds.
