Databricks Architect
SHARVARI INGALEUnited StatesMIDPosted 3h ago
DatabricksSpark SQLPySparkUnity CatalogPower BIAI/BICI/CDData EngineeringAzureSQL ServerData ModelingETL/ELTData WarehousingBI ReportingRelational DatabasesNoSQL DatabasesAI/MLPredictive AnalyticsCloud PlatformsDevOpsAgile/Scrum
About the Role
We are seeking a Databricks Architect to join ComplexCare Solutions. This role involves designing and implementing lakehouse and data warehouse architecture on Databricks, managing batch and streaming pipelines, and ensuring data governance and lineage.
Key Responsibilities
• Scope lakehouse and data warehouse architecture on Databricks, batch and streaming pipelines, enterprise data models, governance and lineage with Unity Catalog, Power BI and AI/BI dashboards, and CI/CD for data engineering.
• Drive adoption of data governance, data quality, lineage, and cataloging frameworks (e.g., Unity Catalog).
• Architect and enable robust data ingestion, transformation, and delivery pipelines (batch and streaming).
• Design systems that support advanced analytics, AI/ML, and self-service BI capabilities on Power BI or AI/BI Dashboards.
• Ensure data accuracy, completeness, consistency, and reliability across platforms.
• Align data architecture with business strategy, reporting needs, and customer-facing analytics capabilities.
• Ensure compliance with enterprise policies, data privacy, security, and regulatory requirements (e.g., HIPAA).
• Ensure production stability, observability, and scalability of data platforms.
• Drive automation, CI/CD, and modern DevOps practices for data engineering.
• Maintain compliance with ComplexCare Solutions (CCS) policies, procedures and mission statement.
• Ensure compliance with CCS standards, HIPAA regulations, and data security requirements.
• Adhere to all confidentiality and HIPAA requirements as outlined within CCS’s Operating Policies and Procedures in all ways and at all times with respect to any aspect of the data handled or services rendered in the undertaking of the position.
Requirements
• 15+ years in software and data architecture (5+ as lead or lead architect role), Spark SQL and PySpark, Databricks agents, AI/BI and Genie, Power BI, SQL Server, Azure, and a healthcare or risk-adjustment background.
• Strong experience with distributed data processing and modern lakehouse technologies (e.g., Spark SQL, PySpark, Unity Catalog, Delta/Iceberg formats, Python, etc.).
• Experience building agents utilizing Databricks Agentic, AI/BI Dashboard, and Genie Spaces capabilities.
• Deep knowledge of data modeling, ETL/ELT design, and data warehousing and BI (Power BI) reporting concepts.
• Experience with relational and NoSQL databases (e.g., SQL Server, PostgreSQL, MongoDB, Cassandra, Hive).
• Experience supporting AI/ML, predictive analytics, and agentic models.
• Background in healthcare, risk adjustment, or regulated domains.
• Hands-on experience with cloud platforms (Azure preferred; AWS or GCP also acceptable).
• Experience with CI/CD pipelines, automated testing, and DevOps practices.
• Experience with Agile/Scrum and modern software delivery methodologies.