Data Engineer
Sudha SriUnited StatesSENIORPosted 13d ago
DatabricksPySparkPythonSQLDelta LakeUnity CatalogMachine LearningGenerative AIAzure
About the Role
We are looking for a Data Engineer with strong hands-on experience in Databricks, PySpark, Python, and SQL. The candidate will build and maintain data pipelines and prepare data for AI and Generative AI applications.
Key Responsibilities
• Build and maintain data pipelines using Databricks, PySpark, Python, and SQL
• Process, clean, and organize data from different sources
• Work with Delta Lake and Unity Catalog to manage and organize data
• Build reliable data pipelines with proper testing, monitoring, and data quality checks
• Prepare data for Machine Learning and Generative AI applications, including embeddings and RAG-based solutions
• Work with cloud services, especially Azure data storage and security tools
• Collaborate with data, application, and AI teams
• Write technical documentation and participate in code reviews
Requirements
• 5+ years of experience in Data Engineering
• Strong practical knowledge of data engineering and data processing
• Experience working with Databricks on Azure or GCP
• Experience with PySpark, Python, and SQL
• Knowledge of Delta Lake and Unity Catalog
• Understanding of Machine Learning and Generative AI applications