Gen AI Data Engineer at NielsenIQ

As a GenAI Data Engineer, you will play a critical role in building the data foundation that powers generative AI applications. You will work closely with software engineers, ML engineers, and product leadership to enable rapid experimentation and drive scalable, production-grade GenAI systems. You will be responsible for designing flexible and scalable data pipelines, enabling retrieval systems, and ensuring high-quality data flows across the GenAI lifecycle—from experimentation to production deployment. Core Responsibilities Data Engineering & Pipeline Development • Design and build scalable batch and near real-time data pipelines using modern data processing frameworks. • Develop robust ETL/ELT workflows for ingesting and transforming structured and unstructured data (documents, PDFs, APIs, logs, etc.). • Leverage cloud-native orchestration and data processing solutions to ensure reliability and scalability. • Implement reusable data frameworks that support rapid experimentation and iteration cycles. • Ensure data quality through validation, schema enforcement, and automated checks.GenAI Data Preparation & Experimentation • Prepare and curate datasets for GenAI use cases including RAG, embeddings, and fine-tuning workflows. • Implement data processing steps like chunking, tokenization, metadata enrichment, and semantic structuring. • Enable fast experimentation loops by supporting dynamic datasets and evaluation pipelines. • Collaborate with engineers and product teams to iterate quickly on features and experiments. • Transition experimental pipelines into production-ready, robust workflows.Vector Databases & Retrieval Systems • Build and maintain embedding pipelines using LLM providers and open-source models. • Design and optimize retrieval systems using cloud-native vector databases and hybrid storage solutions. • Work with relational databases supporting vector capabilities (e.g., PostgreSQL with vector extensions). • Implement and optimize RAG pipelines, including indexing, retrieval, ranking, and refresh strategies. • Manage lifecycle of embeddings, vector indexes, and retrieval datasets.Data Storage & Platform Engineering • Work with cloud-native data platforms and storage solutions (data lakes, lakehouses, object storage). • Design efficient storage schemas for both analytical and retrieval workloads. • Optimize relational and hybrid data stores for low-latency, high-throughput access patterns. • Ensure cost-effective and scalable data storage strategies.Productionization & Scalability • Convert experimental workflows into scalable, reliable production pipelines. • Optimize pipelines for performance, cost, and reliability. • Implement incremental processing, caching, and efficient refresh strategies.Monitoring & Data Observability • Implement monitoring for pipeline health, data freshness, and quality. • Track dataset drift, embedding drift, and retrieval effectiveness. • Build logging, alerting, and observability frameworks for data systems.Collaboration & Cross-Functional Work • Partner closely with engineering teams, product leadership, and data scientists to define and deliver data solutions. • Act as a bridge between rapid experimentation and production engineering. • Contribute to architecture decisions and GenAI data best practices. • Document pipelines, architectures, and data models clearly.Nice-to-Have / Growth Areas • Experience with GenAI frameworks (e.g., LangChain, LlamaIndex, or similar) • Exposure to knowledge graphs and graph-based retrieval approaches • Understanding of data governance, lineage, and cataloging • Experiment tracking and dataset versioningExperience working with multi-modal datasets (text, image, audio)