Infrastructure Engineer
Harshit SrivastavaUnited StatesSENIORPosted 1h ago
TerraformTerragruntDatabricksAWSCloud SecurityPySparkMedallion architectureMLflowAgentic IDEsCI/CDIaC securityGitHub ActionsTerratestCheckovtfsecWizOkta SSO/SCIM
About the Role
We are looking for a seasoned Infrastructure Engineer with deep hands-on expertise in Terraform/Terragrunt, Databricks on AWS, and Cloud Security for a high-priority Corp-to-Corp (C2C) opportunity in San Francisco, CA.
Key Responsibilities
• Design and implement infrastructure using Terraform and Terragrunt
• Manage Databricks on AWS, including workspace provisioning, Unity Catalog, and cluster policies
• Build PySpark data pipelines on Databricks and work with Medallion architecture
• Implement CI/CD and IaC security practices using tools like GitHub Actions, Terratest, and Checkov
Requirements
• 12+ years of overall IT experience, with 7+ years in Infrastructure, Platform, or DevOps engineering
• 3+ years of hands-on production experience with Terraform (Terragrunt strongly preferred)
• 2+ years managing Databricks on AWS
• Deep AWS expertise in Customer-managed VPCs, PrivateLink, IAM, KMS (CMK), S3, Secrets Manager, and CloudWatch
• Experience with PySpark data pipelines, Medallion architecture, MLflow model inferencing, and Agentic IDEs
• Strong CI/CD & IaC security background with tools like GitHub Actions, Terratest, Checkov/tfsec, Wiz, and Okta SSO/SCIM
• Domain experience in Financial Services (Credit decisioning, Risk, or Servicing/Collections) is a plus