Jobs → Data & Analytics → GCP Databricks Architect / Data Engineer

About the Role We are looking for a hands-on GCP Databricks Architect level Data Engineer to build, secure, and scale our core data pipelines within a HIPAA-compliant healthcare data platform. This role focuses on end-to-end pipeline setup, PySpark transformations, and executing workflows using Google Cloud Composer. Key Responsibilities • Pipeline Implementation: Design, build, and deploy batch and streaming data pipelines using Databricks on GCP (PySpark, Spark SQL, Delta Lake). • Workflow Orchestration: Develop, schedule, and monitor production DAGs using Google Cloud Composer (Managed Apache Airflow). • Lakehouse Data Modeling: Implement Delta Lake Medallion architecture, applying optimization techniques like partitioning and Z-ordering. • Healthcare Data Security & Compliance: Ensure strict adherence to HIPAA guidelines for PHI/PII data handling, working with tokenization, encryption, and secure secret management. • Source & Target Integrations: Ingest and egress data across Google Cloud Storage, Cloud SQL / PostgreSQL, and Apigee API gateways. • Governance & Unity Catalog: Configure Unity Catalog structures, catalogs/schemas, and assist with row/column-level security implementations. • CI/CD & Deployment: Package and promote code using Git, GitHub Actions, and Databricks Asset Bundles or Terraform across environments. Requirements • 8+ years of hands-on data engineering experience building production data pipelines. • GCP Infrastructure: GCP, GCS, GKE, Cloud SQL / PostgreSQL, and Google Cloud Composer (Apache Airflow). • Databricks Stack: Databricks on GCP, PySpark, Spark SQL, Delta Lake, Unity Catalog, cluster policies, and job/interactive clusters. • Data Integration & Security: API ingestion patterns (Apigee), JDBC federation, data tokenization, encryption, and HIPAA / PHI / PII compliance frameworks. • DevOps & Scripting: Advanced Python, SQL, Git version control, and CI/CD tools (Terraform or Databricks Asset Bundles).