Data Platform Engineer
Shaik SubhaniUnited StatesSENIORPosted 1h ago
CloudData PlatformGCPAWSTerraformIAMSparkBigQueryTrinoAirflowPythonJavaSQLS3GCSKafkadbtKubernetesAWS LambdaCloud Run
About the Role
We are seeking a Senior Data Platform Engineer to join our team in New York City. This role involves building and operating platform infrastructure for data, ML, and analytics teams, with a focus on data governance, security, data pipeline design, CI/CD, and related tools.
Key Responsibilities
• Build and operate platform infrastructure for data, ML, and analytics teams
• Implement data governance, security, data pipeline design, CI/CD, and related tools
• Manage and optimize core data platform infrastructure, workflow orchestration, and automated data quality frameworks
• Design and implement infrastructure using Terraform or similar IaC tools
• Own platform SLAs, automated alerting, and compute/storage cost optimization
• Implement dataset access control, policy tags, column-level security, row-level security, cost, and quota management
• Troubleshoot, perform root-cause analysis, and handle on-call for shared platform services
Requirements
• 8+ years of total experience, with 5+ years of hands-on Cloud experience in AWS or GCP
• Experience with Terraform or similar IaC tools
• Experience with Spark compute platforms, BigQuery, Trino, job orchestration, and networking
• Strong skills in Python/Java and SQL
• Experience with AWS S3 and GCS
• Experience with data processing, including Spark Jobs, streaming ingestion, data migration, and performance tuning
• Experience with job orchestration, including workflow orchestration (Airflow/Cloud Composer) and automated data quality frameworks
• Experience with data governance, including dataset access control, policy tags, column-level security, row-level security, cost, and quota management
• Experience with collaboration and communication, including coordinating with offshore engineering teams
Benefits
• Long-term contract opportunity
• Opportunity to work on cutting-edge data technologies