Staff / Senior Software Engineer (Agentic Search) – Crawler

Responsibilities

We are representing a confidential AI infrastructure organization seeking a Staff or Senior Software Engineer to lead the development of web-scale crawling and content acquisition systems for a novel search platform designed specifically for AI agents.

In this role, you will build distributed systems that discover, fetch, and continuously refresh content from the open web and large-scale data sources. Your work will enable AI systems to access fresh, verified real-world information through high-performance search APIs.

Your core responsibilities include:

• Design, implement, and operate web-scale crawling systems capable of processing billions of URLs
• Build ingestion workflows for diverse data sources including crawlers, structured feeds, and partner integrations
• Develop crawl scheduling, prioritization, recrawl policies, and freshness strategies to balance coverage and resource efficiency
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
• Ensure reliable operation of high-throughput crawling infrastructure at internet scale
• Define and monitor observability metrics for crawl coverage, freshness, throughput, and content quality
• Optimize resource usage, bandwidth consumption, and infrastructure costs
• Collaborate with indexing and machine learning teams to ensure content quality meets downstream requirements
• Enable safe experimentation with crawling strategies and acquisition policies

What We're Looking For

Required qualifications:

• 5+ years of experience building backend or distributed systems
• Strong expertise in Go or C++
• Proven experience with large-scale distributed systems handling 10,000+ requests per second, billions of URLs, or high-throughput data pipelines
• Deep understanding of web protocols including HTTP, DNS, and TLS
• Hands-on experience with crawling, scraping, and content extraction at scale
• Track record operating production systems and debugging failures in distributed environments
• Strong grasp of scalability, fault tolerance, and resource management principles

Preferred experience:

• Direct web crawling experience
• Building streaming data pipelines and event-driven architectures
• Working with messaging platforms such as Kafka, Pulsar, NATS, or RabbitMQ
• Designing distributed schedulers, queues, and asynchronous processing systems
• Experience with Spark, Flink, Beam, or MapReduce frameworks
• Background in ad tech, social networks, search engines, or other large-scale content platforms

This position is based in London. A coding interview will be part of the selection process. Applicants must be authorized to work in the United Kingdom.

How We Work With You

As a recruiting partner for this confidential client, Get A Job.ai manages the full candidate experience. When you apply through our platform at getajob.ai, one of our specialist recruiters will review your background and conduct an initial screening call. If there's a strong match, we'll submit your profile to our client for consideration and coordinate all interview stages.

Please apply exclusively through Get A Job.ai. Do not attempt to contact the client directly, as this may disqualify your application.

Pay

Compensation details will be discussed during the screening process with our recruiting team.

Equal Opportunity: Get A Job.ai and our clients are committed to fostering inclusive, diverse workplaces. We provide equal employment opportunities without regard to race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity, or any other protected characteristic.