Loading...
SeniorFull-time
Salary Range
$99,450 - $134,550 /year
This salary is estimated based on similar roles. The actual salary may vary.
About the Role
Our client in the Information Technology and Services industry is looking for a Data Engineer to design, build, and optimize streaming data pipelines focused on OpenShift telemetry, observability, and proactive monitoring capabilities.
Key Responsibilities
- →Design, implement, and maintain data pipelines to ingest and process OpenShift telemetry including metrics, logs, and traces at scale.
- →Stream OpenShift telemetry via Kafka and build resilient consumer services for transformation and enrichment.
- →Engineer data models and routing for multi-tenant observability and ensure lineage, quality, and SLAs across the stream layer.
- →Integrate processed telemetry into Splunk for dashboards, alerting, analytics, and proactive insights.
- →Implement schema management using Avro or Protobuf, including governance and versioning for telemetry events.
- →Build automated validation, replay, and backfill mechanisms to ensure data reliability and recovery.
- →Instrument services with OpenTelemetry and standardize tracing, metrics, and structured logging across platforms.
- →Use LLM capabilities to enhance observability through query assistance, anomaly summarization, and runbook generation.
- →Collaborate with platform, SRE, and application teams to integrate telemetry, alerts, and SLOs.
- →Ensure security, compliance, and best practices across data pipelines and observability platforms.
- →Document data flows, schemas, dashboards, and operational runbooks.
Requirements
- ✓Strong expertise in Kafka-based pipelines
- ✓Experience with OpenTelemetry instrumentation
- ✓Knowledge of schema governance using Avro or Protobuf
- ✓Ability to collaborate with platform, SRE, and application teams
- ✓Understanding of observability principles and practices
