Databricks Engineer
Gayatri ThanekarUnited KingdomMIDPosted 13h ago
DatabricksPySparkSQLDelta LakeDatabricks NotebooksJobsWorkflowsUnity CatalogAuto LoaderStructured StreamingAWSS3IAMKMSGlueLambdaLakehouseMedallion ArchitectureBatchIncremental data pipelinesGitCI/CDDistributed data processingTerraformInfrastructure as Code
About the Role
We are looking for a Databricks Engineer to join a leading client in the banking and financial services sector. This contract role requires strong hands-on experience with Databricks and Lakehouse platforms.
Key Responsibilities
• Design and implement data solutions using Databricks and Lakehouse architecture
• Develop batch and incremental data pipelines
• Migrate workloads from AWS-native tools to Databricks
• Utilize PySpark and SQL for data processing
• Work with AWS services including S3, IAM, KMS, Glue, and Lambda
Requirements
• 7–10 years of experience with Databricks and Lakehouse platforms
• Strong hands-on experience with PySpark and SQL
• Experience with Delta Lake, Databricks Notebooks, Jobs, and Workflows
• Knowledge of Unity Catalog, Auto Loader, and Structured Streaming
• Experience with AWS services – S3, IAM, KMS, Glue, and Lambda
• Familiarity with Lakehouse and Medallion Architecture
• Experience with Git, CI/CD, and distributed data processing
• Terraform / Infrastructure as Code experience is an advantage
• Banking or financial services experience is particularly relevant