The Legends Agency
Senior Data Engineer (Kafka Streaming, Spark, Iceberg on Kubernetes)
Build and scale a next-generation real-time data platform with cutting-edge open-source technologies.
100% Remote | R100 000 – R110 000 per month
About Our Client
Our client is a rapidly growing technology-driven organization building high-performance data platforms to enable advanced analytics, AI, and business intelligence. The team operates at the forefront of real-time data processing and distributed systems, leveraging modern cloud-native infrastructure. They foster a culture of technical excellence, continuous learning, and collaboration across multidisciplinary engineering teams.
The Role: Senior Data Engineer
As a Senior Data Engineer, you will design, build, and optimize next-generation data pipelines and platforms. Youll lead the architecture and implementation of scalable, real-time data solutions using Kafka, Spark, and Apache Iceberg deployed on Kubernetes. This is a hands-on, high-impact role within a forward-thinking data engineering team focused on performance, scalability, and innovation.
Key Responsibilities
-
5+ years of professional experience in data engineering or software engineering
-
Design and implement scalable, highly available real-time data pipelines and architectures
-
Build robust ETL and streaming pipelines using Apache Spark (Scala/Python) and Kafka Connect/Streams
-
Develop and manage data lakes using Apache Iceberg with schema evolution and time travel capabilities
-
Deploy and manage distributed data processing services on Kubernetes using containerization best practices
-
Optimize performance and resource usage across Spark jobs, streaming apps, and Iceberg tables
-
Define and uphold engineering best practices including testing, code standards, and CI/CD workflows
-
Mentor junior engineers and contribute to building a high-performing data engineering team
About You
-
5+ years of experience in data engineering or related software engineering roles
-
Advanced proficiency with Apache Spark (batch and streaming)
-
In-depth experience with Apache Kafka (Connect, Streams, or ksqlDB)
-
Hands-on experience with Apache Iceberg, including table evolution and performance tuning
-
Skilled in Python (PySpark) or Scala
-
Experience deploying and managing distributed systems on Kubernetes (Spark Operator is a plus)
-
Solid understanding of data modeling and data warehousing concepts
-
Advantageous: Experience with AWS, Azure, or GCP; familiarity with Flink or Trino
-
Preferred: Bachelors or Masters degree in Computer Science, Engineering, or related field
Originally posted on Himalayas
To apply for this job please visit himalayas.app.
About this role & career path
Working in United States
The United States of America (USA), also known as the United States (U.S.) or America, is a country primarily located in North America. It is a federal republic consisting of 50 states and a federal capital district, Washington, D.C. The 48 contiguous states border Canada to the north and Mexico to the south, with the semi-exclave of Alaska in the northwest and the archipelago of Hawaii in the Pacific Ocean. The United States also asserts sovereignty over five major island territories and various uninhabited islands in Oceania and the Caribbean. It is a megadiverse country, with the world's th
More jobs at The Legends Agency
Keep exploring on Get A Job.ai
Not quite the right fit? Your next opportunity is a click away.
- Browse all jobs
- More jobs by category
- Remote jobs you can do from anywhere
- Research typical pay for this role
- Set a job alert so new matches reach you first
- Upload your resume to apply faster
Hiring instead? Post a job and reach candidates searching right now.