Data Engineer
Vivek P.RemoteSENIORPosted 13d ago
DatabricksSparkPySparkPythonScalaSQLSnowflakeAzure Cloud ServicesADFADLSAzure FunctionsKey VaultLLM Fine-TuningRAGPrompt EngineeringAutomationAirflowGitHubGitHub ActionsCI/CDTerraformInfrastructure AutomationKafka
About the Role
We are looking for a Data Engineer to join our team for a long-term contract. This role involves working with various data engineering tools and technologies.
Key Responsibilities
• Work with Databricks, Spark, and SQL for data processing
• Utilize Azure Cloud Services including ADF, ADLS, and Azure Functions
• Implement DevOps practices with Airflow, GitHub, and CI/CD
• Handle infrastructure automation with Terraform
• Work with Kafka for data streaming
Requirements
• Experience with Databricks, Spark (PySpark), Python/Scala, SQL, and Snowflake
• Knowledge of Azure Cloud Services, ADF, ADLS, Azure Functions, and Key Vault
• Familiarity with DevOps tools like Airflow, GitHub, GitHub Actions, and CI/CD
• Experience with Terraform and Infrastructure Automation
• Understanding of Kafka
Benefits