Data Analyst/Engineer
Kishore SRemoteMIDPosted 2h ago
DatabricksUnity CatalogIceberg REST catalogsSpark Declarative PipelinesPySparkSQLPythonData meshData productsData lake governanceAPI ingestionPower BIData validationData reconciliationData quality monitoring
About the Role
We are looking for a Data Analyst/Engineer to join our team. This role focuses on building and managing data lakes using Databricks and various data cataloging tools.
Key Responsibilities
• Build and maintain data lakes using Databricks
• Work with Unity Catalog, Iceberg REST catalogs, and Spark Declarative Pipelines
• Develop PySpark applications for data processing
• Write complex SQL queries for data analysis
• Implement data mesh architecture and data products
• Design API ingestion processes into raw zones
• Create reports and visualizations using Power BI
• Perform data validation, reconciliation, and quality monitoring
Requirements
• Strong hands-on experience with Databricks for data lake building
• Proficiency with Unity Catalog, Iceberg REST catalogs, and Spark Declarative Pipelines
• Expertise in PySpark and SQL
• Understanding of data mesh, data products, and data lake governance
• Experience with API ingestion into raw zones
• Familiarity with Power BI for reporting and visualization
• Knowledge of data validation, reconciliation, and quality monitoring techniques