Staff Software Engineer – Apache Spark

About This Opportunity

We are representing a confidential enterprise data platform organization seeking a Staff Software Engineer with deep Apache Spark expertise to join their distributed systems team. Our client operates at massive scale—managing data infrastructure that rivals the hyperscalers and serving top companies across nearly every industry. You'll work on core platform components that enable petabyte-scale data processing across thousands of production nodes.

This is a remote position based in the United States. Our talent team at Get A Job.ai is managing the full recruitment process on behalf of our client.

Responsibilities

• Architect and implement next-generation features for enterprise-grade data engineering platforms operating at massive scale across thousands of production nodes
• Contribute directly to Apache Spark as a core contributor, shaping the future of distributed data processing in the open-source community
• Develop high-performance features using Scala, Java, and Python on modern data infrastructure
• Build expertise in SQL planners and optimizers, modern table formats like Apache Parquet and Iceberg, and fault tolerance patterns in large-scale distributed systems
• Conduct root cause analysis and debug complex system-level deployment issues across distributed production clusters
• Improve internal infrastructure and tooling to enhance development, testing, and deployment velocity
• Collaborate with distributed teams and stakeholders to drive technical vision and product delivery

What We're Looking For

Required qualifications:

• 5-7+ years of professional software development experience
• Deep hands-on expertise with Apache Spark internals, including query execution plans (logical and physical), memory management (Tungsten, off-heap allocation, GC tuning), and core data processing flows
• Proven track record leading technical initiatives and delivering complex product enhancements from concept to production
• Strong proficiency in Java, Scala, or other JVM-based languages
• Solid experience designing and developing distributed systems
• Passion for clean code, attention to detail, and focus on software quality and maintainability
• Strong communication skills for effective collaboration across distributed teams
• Demonstrated ability to research, problem-solve, and operate independently
• Growth mindset with eagerness to learn new technologies

Preferred qualifications:

• Experience developing or using Apache Iceberg or related technologies
• Deep experience with large-scale distributed systems design, including scaling, performance optimization, and scheduling
• Experience with SQL planners and optimizers
• Prior contributions to open-source projects

Note: This position is not eligible for visa sponsorship.

How We Work With You

When you apply through Get A Job.ai, our experienced recruiters will review your background and conduct an initial screening conversation. If there's a strong match, we'll submit your profile to our client and guide you through their interview process. We ask that you work exclusively through Get A Job.ai and not contact the client directly—this ensures we can best represent your interests and provide you with guidance at every step.

Pay

Pay details will be discussed during the screening process based on your experience, skills, and location. Compensation is competitive with enterprise technology organizations operating at this scale.

Get A Job.ai is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all candidates we represent, regardless of race, color, religion, sex, national origin, age, disability, veteran status, or any other protected characteristic.