Databricks Architect
Mohammad KhizarUnited KingdomMIDPosted 13h ago
Databricks NotebooksPySpark DataFrame APISQL transformationsDelta LakeDatabricks JobsJob FlowsWorkflowsDeclarative PipelinesS3IAMKMSGlueLambda
About the Role
We are looking for a talented Databricks Architect to join our team based in London or Manchester.
Key Responsibilities
• Design and implement solutions using Databricks Notebooks
• Utilize PySpark DataFrame API for data processing
• Apply SQL transformations to manipulate data
• Work with Delta Lake for data lakehouse architecture
• Manage Databricks Jobs, Job Flows, and Workflows
• Develop Declarative Pipelines using dpx files and pipeline definitions
• Integrate with AWS services like S3, IAM, and KMS
• Collaborate with Glue and Lambda for data processing
Requirements
• Strong hands-on experience with Databricks Notebooks
• Proficiency in PySpark DataFrame API
• Experience with SQL transformations
• Knowledge of Delta Lake
• Familiarity with Databricks Jobs, Job Flows, and Workflows
• Understanding of Declarative Pipelines
• Experience with AWS services (S3, IAM, KMS)
• Experience with Glue and Lambda