Jobs → Data & Analytics → Databricks Architect

About the Role We are looking for an experienced Databricks Architect with strong expertise in AWS, data engineering, and Lakehouse architecture to design and optimize scalable enterprise data platforms. Key Responsibilities • Design end-to-end Databricks Lakehouse architectures on AWS • Build scalable ETL/ELT pipelines using PySpark, Spark SQL, and Delta Lake • Implement Medallion Architecture (Bronze, Silver, Gold) • Optimize performance using partitioning, caching, Z-ORDER, OPTIMIZE, and file compaction • Design low-latency data serving solutions using APIs and serving-layer technologies • Implement streaming and incremental data processing using Structured Streaming • Ensure data security, governance, lineage, and quality using Unity Catalog and RBAC • Optimize cluster sizing, autoscaling, workload performance, and cloud costs • Lead architectural reviews, PoCs, and provide hands-on technical guidance to engineering teams Requirements • 12+ years of overall IT experience • 7+ years in Data Engineering / Data Architecture • 3+ years of hands-on Databricks experience • Strong expertise in Apache Spark, PySpark, SQL, and distributed data processing • Deep understanding of Delta Lake, Auto Loader, OPTIMIZE, and Z-ORDER • Strong AWS experience: S3, Glue / Glue Catalog, IAM, Lambda, API Gateway, and Kinesis • Experience with Databricks Workflows, REST APIs, CI/CD, Git, and Infrastructure as Code • Knowledge of Terraform, Unity Catalog, data governance, and performance tuning Good to Have • Databricks certifications • MLflow, Feature Store, or MLOps experience • Experience in BFSI, healthcare, or other regulated industries