Lead Data Engineer (Azure PaaS & Product Engineering) (10+ Years of Exp) at 3Pillar

3Pillar

India·1w ago

LeadFull-timePythonAzureKubernetesDockerTerraformREST APIMicroservicesCI/CDGitAI

Salary Range

$30,600 - $41,400 /year

Estimated

This salary is estimated based on similar roles. The actual salary may vary.

About the Role

We are building a customer-facing data platform from the ground up. The platform requires real-time data ingestion, high availability, and integration with Generative AI. You will lead architecture decisions on Azure, sitting between Backend Engineering and Data Engineering.

Key Responsibilities

→Design event-driven ingestion architectures. Select appropriate Azure compute (Functions, Durable Functions, containers) based on workload characteristics.
→Design OLTP schemas for sub-second application retrieval, not just offline reporting.
→Architect data flows for RAG workflows. Ensure unstructured text is processed and indexed for vector search.
→Build reusable Python frameworks. Enforce unit testing, CI/CD, and code reviews within the data team.
→Own synchronization between operational databases (SQL/NoSQL) and analytical workloads. Design for consistency, replay capability, and clear separation between serving and analytical layers.

Requirements

✓10+ years in IT, with 5+ years as a Lead or Architect.
✓Software engineering background (APIs, microservices) before moving into data. You understand that cloud data engineering is software development.
✓Expert Python skills for backend processing, API integration, and automation. Clean, testable, production-grade code.
✓Hands-on Azure experience: Functions (Consumption/Flex), Event Grid, Event Hubs. You understand tradeoffs between serverless, container-based, and dedicated compute.
✓Experience building pipelines that feed LLMs or vector databases.
✓Lakehouse architecture: medallion patterns, Delta Lake semantics, separation of storage and compute. Experience with Microsoft Fabric or Azure Databricks.
✓Data modeling for both transactional systems and analytics. You know the difference between designing for an app vs. a data lake.