Data Engineer
Rahul S.United StatesSENIORPosted 13d ago
CDPGCPCloud SpannerBigQueryPub/SubDataflowApache BeamPythonJavaSQLPySparkDatabricksSparkDelta LakeSnowflakeAzure Data FactoryEvent HubsCosmos DBKafkaCloud RunDataprocComposerAirflowTerraformCI/CD
About the Role
We are looking for a Data Engineer with strong expertise in Customer Data Platforms (CDP), Google Cloud Platform, Cloud Spanner, and real-time data engineering to build scalable pipelines supporting 100M+ customer profiles, low-latency APIs, and AI-powered customer experiences.
Key Responsibilities
• Build real-time and near-real-time pipelines using Pub/Sub, Dataflow, and Apache Beam
• Ingest and process customer, clickstream, care, store, and network data into Cloud Spanner and BigQuery
• Optimize Cloud Spanner schema, primary keys, indexes, batching, and hotspot handling for low-latency reads
• Develop reusable CDC, SCD Type 2, and late-arriving-data processing patterns
• Migrate and modernize existing Azure data pipelines to GCP, leveraging Databricks, Snowflake, and Azure Data Factory
• Build data feeds for Customer 360, knowledge graphs, GraphQL APIs, and MCP servers supporting AI agents
• Implement data quality, lineage, reconciliation, security, monitoring, and SLOs
• Use Terraform and CI/CD to deploy and manage cloud-native data services
• Collaborate with product, analytics, and AI/ML teams while mentoring junior engineers
Requirements
• 15+ years of Data Engineering experience
• FAANG background mandatory
• 3+ years of production experience with GCP data platforms
• 3+ years of real-time / near-real-time streaming pipelines
• 3+ years of BigQuery
• 1+ year of Cloud Spanner
• 4+ years of data modeling, ETL/ELT, SQL, and Python
• 3+ years of Databricks, Spark, and Delta Lake
• 2+ years of Azure and Snowflake
• Hands-on experience with Pub/Sub, Dataflow, Apache Beam, and Kafka
• Strong PySpark, CDC, SCD Type 2, and clickstream ingestion
• Experience with Terraform, CI/CD, orchestration, and data quality