Data Engineer
Palantir AI SolutionsRemoteSENIORPosted 7h ago
PythonSQLPySparkDelta LakeDatabricksAEPAPIsCloud storage
About the Role
We are looking for a Data Engineer to build and maintain data pipelines connecting Databricks and AEP. This role involves developing scalable ingestion and transformation workflows and supporting data movement between enterprise data platforms.
Key Responsibilities
• Build, maintain, and troubleshoot data pipelines connecting Databricks and AEP
• Develop scalable ingestion and transformation workflows using Python, SQL, PySpark, and Delta Lake
• Map and transform customer, behavioral, transactional, and campaign data into AEP Experience Data Model (XDM) schemas
• Support batch and streaming data movement between enterprise data platforms and AEP
• Collaborate with marketing, analytics, and technology teams to support customer profiles, audience segmentation, personalization, and campaign activation
• Monitor pipeline performance, data quality, reliability, and freshness
• Investigate and resolve production issues
• Follow data governance, privacy, consent, and access-control requirements when handling customer data
• Maintain documentation and support knowledge transfer