Senior Software Engineer II [Lead Data Engineer / Mumbai OR Bangalore] at TheNielsenCompany

As the Lead Data and AI Engineer, you are the technical architect for Nielsen’s GenAI and data evolution in the region. You will bridge the gap between high-level strategy and functional execution by leading the setup, definition, and industrialization of our Databricks ecosystem. This role is not just about writing code; it is about building a sustainable, governed engineering foundation that transitions experimental AI into enterprise-grade solutions. You will act as the "Technical Commander" for the Databricks environment, ensuring every solution is outcome-driven and architecturally sound. Core Goals & Responsibilities • Databricks Ecosystem Leadership: Lead the technical definition and setup of the Databricks environment. You are responsible for ensuring scalability and maintaining the integrity of the Unity Catalog for global data governance. • End-to-End Engineering: Architect and build robust pipelines using common open-source data formats, ensuring they are optimized for both traditional analytics and emerging AI workloads. • Operational Excellence: Partner closely with CloudOps and DevOps to ensure 99.99% reliability and seamless CI/CD integration. • Functional Collaboration: Engage directly with Finance, HR, and Product owners to understand business requirements and translate them into high-impact technical solutions. • Community & Mentorship: Build and lead a community of developers within the Databricks environment. You will provide technical guidance, unblock delivery friction, and ensure the team builds with "Strategic Foresight" regarding cloud costs (FinOps).  Expertise & Technology Stack • Core Engineering: 8–10+ years of deep experience in data engineering and architectural design. • Frameworks & Languages: Expert-level proficiency in Python, Heavy SQL, and dbt for transformation layers. • AI & Orchestration: Hands-on experience with LangChain and vector database integration for GenAI applications. • Databricks Mastery: Deep knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.Analytical Thinking: Ability to challenge status-quo assumptions and provide a pragmatic pivot when technical risks arise.