Data Engineer
Xidthiprom R.RemoteMIDPosted 11h ago
SQLPythonPySparkKafkaSpark Structured StreamingDatabricksApache SparkAirflowAzure Data FactoryGCPBigQueryDataflow
About the Role
We are looking for a Data Engineer to join our team, focusing on real-time data and streaming technologies. This role involves developing and maintaining data pipelines for both batch and real-time processing.
Key Responsibilities
• Develop and maintain Data Pipeline for both Batch ETL/ELT and Real-time / Near Real-time Streaming
• Connect and manage data from multiple sources such as Database, API, and Event Streams
• Perform Data Transformation, Data Validation, and Data Reconciliation
• Design and improve Data Flow / Data Integration for Data Lakehouse and Data Warehouse
• Work with Data Architect, Developer, and customer team from Mapping, Testing, SIT/UAT to Production
Requirements
• 3–5 years of experience as a Data Engineer
• Proficiency in SQL, Python / PySpark
• Experience in Real-time Data Streaming such as Kafka, Spark Structured Streaming, or Pub/Sub
• Experience with Databricks, Apache Spark, Airflow, or Azure Data Factory
• Experience with GCP, BigQuery, Dataflow will be considered as a plus
• Understanding of Data Integration, ETL/ELT, and database
• Ability to explain Data Flow and Architecture, communicate, and coordinate effectively