Principal Engineer at Nexus Cognitive

ABOUT NEXUS At Nexus, we help organizations turn complex data into clear, actionable insight — so the hard work behind artificial intelligence, analytics, and cloud infrastructure doesn’t slow innovation down… it accelerates it. We’re a team of builders, problem-solvers, and collaborators who believe in moving fast, learning continuously, and doing great work together. Our role is to handle the heavy lifting of modern data platforms, governance, and cloud transformation so our clients can focus on making confident, strategic decisions that move their businesses forward. We’re proud of the culture we’re building across the U.S. and India — one rooted in curiosity, ownership, collaboration, and care — and we’re excited to keep growing with people who want to do meaningful work alongside genuinely great teammates. Role Summary The Principal Engineer is the highest individual contributor on the NX1 platform engineering team. They hold architectural authority across the NX1 codebase and are the primary technical decision-maker below the Head of Engineering. They define how NX1 components are built, integrated, and evolved — with deep ownership of the platform's most complex and strategically important subsystems. The NX1 stack is built on Apache Spark, Trino, Iceberg, Ranger, Gravitino, Keycloak, Airflow, DataHub, and Debezium — all running on Kubernetes and OpenShift. The Principal Engineer is expected to have significant depth across most of these components and to lead the engineering team in building production-grade, enterprise-hardened implementations of each. They also serve as the escalation point for the most complex issues surfaced through L3 support and Forward Engineering deployments. Key Responsibilities • Own the architecture of NX1's core platform: defining how Spark, Trino, Iceberg, Ranger, Gravitino, and Keycloak are integrated, versioned, and deployed together • Lead the design of NX1's multi-tenant security model — Ranger policy enforcement, Keycloak-based JWT authentication, Gravitino credential vending, and YuniKorn-based namespace isolation on Kubernetes • Drive the Iceberg catalog strategy: Gravitino as Iceberg REST catalog, namespace/tablelifecycle management, S3 credential vending, and Spark/Trino catalog configuration • Author technical design documents (TDDs) for major features and integration efforts; lead cross-team technical reviews and drive adoption of architectural decisions • Own the engineering team's integration posture with open-source upstream projects —evaluating major version upgrades, managing patches, and leading contribution efforts whereNX1 has custom extensions • Lead technical investigation of the most complex escalations from Forward Engineering —issues involving Spark/Trino query failures, Ranger policy conflicts, Keycloak token issues, or Gravitino catalog inconsistencies • Define NX1's Kubernetes operator strategy: how platform components are deployed, upgraded, and managed via Helm and Terraform across customer environments • Mentor Senior Software Engineers and set engineering standards for code quality, testing,observability, and release practices • Partner with the Head of Engineering on technical roadmap planning, build vs. integrate decisions, and OSS component selection • Interface directly with enterprise customers on the most complex architectural questions raised during Solutions Architect-led engagements Qualifications & Experience • 10+ years software engineering experience with at least 3 years at staff or principal engineer level on distributed data systems • Expert-level knowledge of Apache Spark — including Spark on Kubernetes, the Spark SQL engine, custom extensions, and performance tuning at scale • Deep experience with Trino (formerly PrestoSQL): connector architecture, query planning, cost-based optimization, and integration with Hive Metastore or Iceberg REST catalog • Hands-on production experience with Apache Iceberg: table format internals, partitioning strategies, compaction, snapshot expiry, and multi-engine compatibility (Spark + Trino) • Strong experience with enterprise authentication and authorization: Keycloak (OIDC/JWT), Apache Ranger (policy engine, service definitions, plugins), and LDAP/Active Directory integration • Experience with Apache Gravitino or comparable metadata/catalog platforms: catalog API design, Iceberg REST catalog implementation, and credential vending patterns • Kubernetes and OpenShift expertise: operators, CRDs, RBAC, network policies, and multi-tenant namespace management; experience with YuniKorn or equivalent schedulers is a strong plus • Proficiency in Java and/or Scala for Spark/Trino work; Python for tooling, automation, and data engineering integrations • Experience with Helm chart authoring, Terraform module development, and GitOps-based platform lifecycle management • Track record of leading engineering teams through complex open-source integrations, major version migrations, and enterprise customer deployments WHY YOU’LL LOVE WORKING AT NEXUS At Nexus, we value people who want to grow — and support each other while doing so. You can expect: - A collaborative team culture built on curiosity and respect - Challenging work where your contributions clearly matter - A leadership team that invests in learning and development - The opportunity to work at the intersection of cloud, data, and AI innovation READY TO APPLY? If this role sounds like a great fit — or even close to one — we’d love to hear from you. We know that no candidate checks every single box, and we’re excited to meet people who bring curiosity, talent, and a desire to build meaningful work together.