Data Engineer
HEITS Pvt. LtdRemoteMIDPosted 13h ago
DatabricksDelta LakeIcebergDelta Live TablesUnity CatalogueYAMLAirflowdbtGreat ExpectationsCollibraApache SparkPrestoAmazon EMR
About the Role
We are looking for skilled Data Engineers and Data Architects for contract opportunities with one of our esteemed clients. This role requires experience in designing and building robust data pipelines and complex job/workflow orchestration patterns.
Key Responsibilities
• Design and develop data pipelines with Databricks, including Delta Lake, Iceberg, Delta Live Tables, and Unity Catalogue
• Build robust, scalable data pipelines with batch, micro-batch, and streaming data ingestion and processing patterns
• Implement complex job/workflow orchestration patterns using Databricks jobs/workflows or other orchestration tools like Airflow, dbt
• Build data quality and pipeline audit/observability solutions
Requirements
• 5+ years of data engineering experience, with at least 2 years of hands-on data pipeline design and development experience with Databricks
• Proficiency in Big Data Platforms: Apache Spark, Presto, Amazon EMR
• Experience in building robust data quality, pipeline audit/observability solutions
• Shift Timings: 2:30 PM to 11:30 PM IST
• Contract Duration: 6 Months (extendable based on client requirement)