Senior Software Engineer II (Databricks Engineer) at TheNielsenCompany

Strategic Mandate As a Senior Data and AI Engineer II, your mandate is to bridge the gap between architectural design and hands-on execution. Reporting into the Lead Data and AI Engineer, you will be a key contributor to the AI Braintrust, focusing on the technical industrialization of our Databricks environment. This role is built for a "Strategic Executor"—someone who understands the functional "how" of scaling AI pipelines while maintaining a sharp focus on the "why" behind business requirements from Finance, HR, and Product stakeholders. Core Goals & Responsibilities • Databricks Engineering: Execute the end-to-end build of data solutions within the Databricks ecosystem, ensuring all pipelines are optimized for performance and cost-efficiency (FinOps). • Unity Catalog Implementation: Assist in the definition and rigorous maintenance of the Unity Catalog to ensure data governance, residency, and security protocols are met across all workflows. • Modern Tooling & Frameworks: Develop and maintain transformation layers using dbt, and leverage Python and Heavy SQL to build resilient data structures. • AI Orchestration: Implement GenAI patterns using LangChain and open-source data formats to transition experimental models into production-ready features. • Collaborative Delivery: Work in lockstep with CloudOps and DevOps to ensure seamless CI/CD integration and 99.99% system reliability. • Stakeholder Engagement: Partner with owners in Finance and HR to ensure data outputs are accurate, actionable, and aligned with departmental goals. Expertise & Technology Stack • Experience: 5+ years of progressive experience in data engineering, with a focus on cloud-native architectures. • Technical Proficiency: * Advanced Python and SQL skills. • Hands-on experience with dbt for data modeling. • Exposure to AI frameworks such as LangChain. • Cloud Infrastructure: Familiarity with AWS environments and automated deployment patterns. • Problem Solving: A pragmatic approach to engineering; you focus on building sustainable solutions rather than "over-engineering" for the sake of complexity.