Loading...
SeniorFull-timeKubernetesData ScienceCI/CD
Salary Range
$79,560 - $107,640 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
We are looking for a Data Platform Engineering / DataOps professional to join our team. This role focuses on creating production pipelines and strengthening the ecosystem using standards, quality controls, governance measures, and observability, so that product teams can deliver reliable data more quickly.
Key Responsibilities
- →Migrate multi-technology architectures to Databricks + Unity Catalog + dbt
- →Evaluate existing flows (batch/stream, orchestrations, models, cost/performance) and define target models (Delta Lake, Databricks Jobs/Workflows, dbt models)
- →Rewrite pipelines (Kubernetes/Airflow/Spark/SQL) and replatform models in dbt by applying naming conventions, tests, and documentation
- →Integrate assets into Unity Catalog (RBAC/ACL, lineage, data masking/PII), structure catalogs/schemas, and manage permissions
- →Implement robust and fault-tolerant systems for data ingestion and processing
- →Strengthen data CI/CD (dbt tests in PR, linting, schema checks) and orchestration with retry/backoff, dependencies, and SLA compliance
- →Deploy end-to-end observability (pipeline metrics, freshness, test coverage) and alerts (e.g., Datadog) with playbooks and MTTR targets
- →Define and monitor data contracts and SLOs with consumer teams and ensure compliance
- →Keep documentation up to date (catalog, metric definitions, change logs) and create standardized runbooks for data audit and deployment
- →Optimize Delta tasks/tables (partitioning, Z-Order, OPTIMIZE, autoscaling); monitor costs/performance (budget, unit profitability per pipeline)
- →Collaborate with various teams, including data science, infrastructure, software, product, and consulting teams, to understand their data needs and provide solutions
- →Work with Client Analytics Engineers to translate retailer needs into reliable and maintainable models
- →Share best practices (dbt models, integration guides, quality checklists)
- →Train team members in SQL querying in Databricks and lead updates during Data Domain sessions
Requirements
- ✓Experience with Databricks, Unity Catalog, and dbt
- ✓Knowledge of Kubernetes, Airflow, Spark, and SQL
- ✓Understanding of CI/CD principles and practices
- ✓Familiarity with observability tools like Datadog
- ✓Strong problem-solving and analytical skills
- ✓Excellent communication and collaboration abilities
- ✓Ability to work in a fast-paced environment and manage multiple projects simultaneously
