Senior Staff Software Engineer, Data Infrastructure at Instacart

We're transforming the grocery industry At Instacart, we invite the world to share love through food because we believe everyone should have access to the food they love and more time to enjoy it together. Where others see a simple need for grocery delivery, we see exciting complexity and endless opportunity to serve the varied needs of our community. We work to deliver an essential service that customers rely on to get their groceries and household goods, while also offering safe and flexible earnings opportunities to Instacart Personal Shoppers. Instacart has become a lifeline for millions of people, and we’re building the team to help push our shopping cart forward. If you’re ready to do the best work of your life, come join our table. Instacart is a Flex First team There’s no one-size fits all approach to how we do our best work. Our employees have the flexibility to choose where they do their best work—whether it’s from home, an office, or your favorite coffee shop—while staying connected and building community through regular in-person events. Learn more about our flexible approach to where we work. About the Role Instacart's backend systems serve millions of customers every year, processing petabytes of data daily to power everything from personalized grocery recommendations to real-time fraud detection and advertiser measurement. As Senior Staff Software Engineer on the Data Infrastructure team, you will set the technical direction for the data platform that underpins the entire company’s data strategy — from the storage and compute layer to streaming infrastructure, analytics tooling, and governance systems. This is a role for a systems thinker who thrives at the intersection of deep technical depth and company-level strategic impact. You will define the multi-year architecture roadmap for our core data platform, drive major platform investment decisions, and operate as a thought leader both internally across engineering leadership and externally within the data infrastructure community. Your work will directly influence how Instacart makes decisions at scale and shape the platform economics of one of the most data-intensive technology companies in grocery commerce. About the Team The Data Infrastructure team builds and operates the systems that power Instacart’s data ecosystem — a modern data lakehouse built on Apache Iceberg, a multi-engine compute platform spanning stream processing and analytical query engines, and self-serve tooling that enables Product, Data Science, ML, Ads, Finance, and other engineering teams to move fast with data. We balance two mandates: maintaining a highly reliable production data platform serving the business today, and architecting the infrastructure that will serve it for the next three to five years. We operate in a world of significant scale and material infrastructure spend, where architectural decisions carry both technical and financial consequences. Some of the core technologies in our stack include: Apache Iceberg, Apache Flink, Trino, ClickHouse, Apache Kafka, Apache Spark, Snowflake, Databricks, Confluent, Airflow, dbt, Delta Lake, Scala, Python, Postgres, and AWS. The team operates with a high degree of ownership and autonomy. You will work closely with engineering leadership, data science, ML platform, ads infrastructure, finance engineering, and senior stakeholders across the organization. ABOUT THE JOB What You’ll Do • Define and Drive Data Infrastructure Vision: Own the multi-year technical vision and roadmap for Instacart’s core data platform (storage, compute, streaming, orchestration, analytical serving). Translate company data strategy (monetization, federated access, real-time) into a coherent, actionable architecture plan. Align with leadership and proactively evolve the architecture for scale, maturity, and cost. • Lead Platform Strategy (Build, Buy, Ownership): Architect the ownership strategy for the data platform, determining build vs. buy (including managed services vs. open-source self-hosting). Lead technical/business case evaluations, full cost-benefit modeling, and risk analysis for major investments. Design phased migrations to ensure reliability while achieving long-term independence and cost efficiency. • Own the Data Lakehouse Foundation: Drive the architecture and delivery of the open lakehouse, including unified table format, compute engine portfolio, and storage governance. Expand multi-engine compute (interactive, batch, stream processing). Define standards for data storage, access, governance, and sharing to enable compute portability and prevent lock-in. Ensure reliable scaling without proportional cost increase. • Drive Real-Time and Streaming Infrastructure: Own the architecture for streaming data, event-driven pipelines, stream processing, and real-time serving for critical use cases (Ads, Fraud, ML). Make principled decisions on deployment models balancing cost, availability, and operational maturity. • Pioneer AI-native Data Infrastructure Engineering: Lead the adoption, application, and cultural integration of AI/LLM tools across the data platform lifecycle, setting a high standard for AI-augmented workflows, driving high-leverage opportunities from automation to cost optimization, and partnering with other teams to embed AI-powered capabilities into the platform itself. • Elevate Engineering Excellence: Serve as the senior technical voice, setting standards for system design and reliability. Lead architecture reviews. Mentor staff/senior engineers, fostering ownership and execution. Be a visible engineering leader, contributing to hiring and cross-org alignment. • Partner Deeply with Stakeholders: Collaborate with Data Science, ML Platform, Ads Infra, Product Eng, Finance Eng, and Security to translate needs into reliable, self-serve infrastructure. Represent Data Infra in architectural forums, ensuring decisions support business priorities (monetization, compliance, AI). Clearly communicate complex trade-offs to technical and executive audiences. Minimum Qualifications • 10+ years of software engineering, focused on data infrastructure or distributed systems at scale.Sets technical direction for large-scale data platforms, defining multi-year architecture roadmaps and influencing strategy. Experience in high-growth, data-intensive environments with significant infrastructure scale and spend. • Expertise in modern data lakehouse architectures, open table formats (Iceberg, Delta Lake, Hudi), and compute/storage trade-offs, in distributed query/compute systems (Trino, Spark, ClickHouse, etc.) for performance tuning and production reliability and event-driven infrastructure (Kafka, Flink, etc.) • Proven track record owning and executing major infrastructure platform transitions, including build vs. buy, migration design, and risk management. • Experience building compelling business cases for infrastructure investments, including cost-benefit analysis and TCO modeling. • Exceptional technical communication for clear architecture documents, strategy memos, and proposals to drive leadership alignment. Strong ownership, comfort with ambiguity, and organizational influence to drive large, multi-team initiatives from concept to production. Preferred Qualifications • Familiarity with data governance, compliance frameworks (SOX, CPRA, GDPR), and designing governance controls into the platform architecture. • Experience with FinOps and data platform cost optimization, including managing multi-million dollar infrastructure budgets and negotiating vendor contracts. • Deep knowledge of SQL and strong proficiency in Python or Scala for systems-level work. • Experience with orchestration systems (e.g., Apache Airflow) and data transformation pipelines (e.g., dbt) in large-scale production environments. • Track record of building and growing high-performing data infrastructure teams. • Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience. #LI-Remote Instacart provides highly market-competitive compensation and benefits in each location where our employees work. This role is remote and the base pay range for a successful candidate is dependent on their permanent work location. Please review our Flex First remote work policy here. Offers may vary based on many factors, such as candidate experience and skills required for the role. Additionally, this role is eligible for a new hire equity grant as well as annual refresh grants. Please read more about our benefits offerings here. For US based candidates, the base pay ranges for a successful candidate are listed below. CA, NY, CT, NJ$304,000—$321,000 USDWA$292,000—$308,000 USDOR, DE, ME, MA, MD, NH, RI, VT, DC, PA, VA, CO, TX, IL, HI$279,000—$294,500 USDAll other states$254,000—$268,000 USD