Web Crawler

What is a web crawler?

A web crawler fetches pages from the internet and stores their content for downstream indexing. Downloading HTML is the easy part. The real challenge is doing it at a billion-page scale while staying polite to target servers, deduplicating URLs across petabytes of already-visited content, and keeping pages fresh without re-crawling the entire index on every cycle. This question tests distributed queues, Bloom filters, scheduling algorithms, and rate limiting all at once, which is exactly why interviewers reach for it.

I'd design this as a periodic pull-based system rather than real-time. A real-time crawler would require a push architecture where publishers notify you via WebSub (PubSubHubbub) or sitemap ping APIs, and you process page updates as events arrive. That changes the URL Frontier from a scheduled priority queue into a streaming ingestion pipeline (a Kafka topic with event-time processing). For a general-purpose crawler covering billions of arbitrary pages, the push model is impractical: most pages don't implement WebSub. The periodic model with adaptive re-crawl intervals is the right default, and real-time supplements (sitemap pings, RSS polling) layer on top without replacing the core architecture.

Functional Requirements

Core Requirements

1. Crawl all pages reachable from a seed list of URLs.
2. Store the raw HTML and extracted links for downstream indexing.
3. Discover new URLs continuously and re-crawl stale pages on a schedule.

Below the Line (out of scope)

• Full-text search indexing and ranking (downstream system)
• JavaScript rendering (Puppeteer/headless browser tier)
• Login-gated content crawling
• Media file (image/PDF) extraction

Full-text indexing is a downstream concern that consumes the raw HTML this crawler produces. If in scope, I'd add an Index Service that reads from the HTML store, tokenizes content, computes TF-IDF or BM25 scores, and writes to an inverted index store. The crawler and the indexer share exactly one interface: the raw HTML storage layer, so both can evolve independently.

JavaScript rendering requires a headless browser tier (Puppeteer, Playwright) that is 10 to 50 times more expensive per page than a plain HTTP fetch. If in scope, I'd route detected JavaScript-heavy pages to a separate rendering queue with much lower throughput (50 to 100 pages per second instead of 1,000) and pass the rendered HTML back into the main pipeline.

Login-gated content requires per-site credential management, session handling, and CAPTCHA handling. It is sufficiently different from general-purpose crawling that I'd build it as a separate, targeted crawl service rather than generalizing the main crawler.

Media extraction (images, PDFs) is excluded because the storage and processing requirements are fundamentally different: blob storage, OCR pipelines, image recognition. The crawler still encounters these URLs during link extraction, but it discards the binary payload and stores only the URL reference.

The hardest part in scope: URL deduplication at petabyte scale is the single most challenging problem here. At 1 billion pages, a memory-resident hash set requires hundreds of gigabytes of RAM. A database lookup per URL becomes a write-path bottleneck at 1,000 pages per second. Getting deduplication right determines everything else about the crawler's performance.

Non-Functional Requirements

Core Requirements

• Scale: 1 billion pages crawled total; 100 million new or updated pages per day.
• Throughput: 1,000 pages per second sustained (86.4 million pages per day at full capacity).
• Politeness: Maximum 1 request per domain per second; 1 per 10 seconds for sensitive or slow-responding domains.
• Deduplication: No URL crawled twice within a single crawl cycle.
• Freshness: High-change pages (news sites, live feeds) re-crawled within 24 hours; low-change pages within 30 days.
• Storage: Raw HTML stored durably; estimated 1 TB per 1 million pages = 1 PB total at full scale.
• Availability: The crawler runs continuously; transient failures must not lose queued URLs.

Below the Line

• Sub-millisecond deduplication latency (seconds-scale latency is acceptable for the URL queue)
• Exactly-once crawl guarantees (at-least-once with idempotency is sufficient)

Read/write ratio: This system is almost entirely writes. For every URL fetched and stored, there is 1 deduplication check, 1 HTML write to object storage, and several URL queue operations. The only significant read workload is the downstream indexer reading from HTML storage. Design every component for write throughput, not read latency. This is one of the few large-scale systems where you can deprioritize read optimization almost entirely.

Core Entities

• CrawlJob: A top-level crawl task with a seed URL list, status, and configuration (depth limit, domain scope, re-crawl enabled flag).
• URLFrontier: A prioritized queue entry representing a URL to be crawled, with a scheduled crawl time and a numeric priority score.
• CrawledPage: The stored result of one crawl: raw HTML, extracted outbound links, HTTP status code, crawl timestamp, and content hash.
• DomainPolicy: The cached per-domain rules: parsed robots.txt directives, crawl delay setting, last-crawl timestamp, and domain authority score.
• URLFingerprint: A compact record (canonical URL string and its hash) used for Bloom filter deduplication lookups.

Full schema and indexing strategy are deferred to the deep dives. These five entities are enough to drive the API and High-Level Design.

API Design

A web crawler is primarily internal, but operator APIs are needed for seed submission and observability.

FR 1: Submit seed URLs to start a crawl:

POST /v1/crawl/seeds
Body: {
  urls: ["https://example.com", "https://news.ycombinator.com"],
  config: { max_depth: 5, scope: "domain", recrawl_enabled: true }
}
Response: { job_id: "cj_abc123", queued_count: 2, status: "queued" }

POST because this creates a new CrawlJob. The config block lets callers scope the crawl to a single domain or the full reachable web, and enables or disables freshness-based re-crawling per job. The job_id is returned for status polling; callers do not wait synchronously for the crawl to complete.

FR 2: Check the crawl status of a specific URL:

GET /v1/crawl/status/{encoded_url}
Response: {
  url: "https://example.com",
  status: "crawled",
  last_crawled_at: "2026-03-29T12:00:00Z",
  next_scheduled_at: "2026-03-30T06:00:00Z",
  http_status: 200,
  content_hash: "sha256:abc..."
}

The URL is URL-encoded in the path. next_scheduled_at lets downstream integrations know when refreshed content will be available. content_hash reveals whether a re-crawl produced any actual change, which matters for incremental indexers.

FR 3: List active crawl jobs:

GET /v1/crawl/jobs?status=running&cursor=eyJ0c...&limit=20
Response: {
  jobs: [
    { job_id: "cj_abc", seed_count: 2, pages_crawled: 15420, status: "running", started_at: "..." }
  ],
  next_cursor: "eyJ0c..."
}

Use cursor-based pagination because the job list is a time-ordered stream. Offset pagination skips jobs when new crawls start mid-page. Filter by status (queued, running, completed, failed) to limit result set sizes in production.

High-Level Design

1. Basic crawl loop: frontier, fetcher, and storage

The core pipeline: dequeue a URL, fetch the page, parse links, store HTML, and enqueue discovered URLs.

This satisfies FR 1 and FR 2 end-to-end on a small seed set. It has no politeness enforcement, no deduplication, and no priority control. Establishing correctness first gives us a clear baseline before adding complexity.

Components:

• Seed Submitter: Operator API that pushes seed URLs into the URL Frontier after job creation.
• URL Frontier Queue: A persistent FIFO queue (Kafka or Redis sorted set) holding URLs pending crawl.
• Fetcher Service: Dequeues URLs, sends HTTP GET to target servers, receives raw HTML, and extracts outbound links.
• HTML Store: Object storage (S3) for raw HTML, keyed by URL hash for content-addressed lookup.
• Crawl DB: PostgreSQL tracking crawl status per URL (queued, crawled, failed, skipped).

Request walkthrough:

1. Seed Submitter calls POST /v1/crawl/seeds and the API writes seed URLs into the URL Frontier Queue.
2. Fetcher Service dequeues the next URL and sends HTTP GET to the target server.
3. Fetcher writes raw HTML to HTML Store, keyed by the SHA-256 hash of the URL.
4. Fetcher extracts all <a href> links from the HTML and checks Crawl DB: for each link not yet queued, enqueues it into the URL Frontier.
5. Fetcher updates Crawl DB: mark the URL as crawled, record http_status and crawl_timestamp.

This covers the basic pipeline. I like to start interviews with this exact diagram because it proves you understand the core loop before adding any optimization layers. The fetcher hammers the same domain without throttling, and re-enqueues already-visited URLs, so the frontier grows unboundedly. Both problems are addressed next.

2. Politeness enforcement

A fetcher that fires requests without throttling will hit the same domain dozens of times per second, triggering rate-limit responses or outright IP bans.

Politeness has two components: respecting robots.txt (which URLs the site disallows) and enforcing per-domain crawl delays (how fast we hit a given server). I'd treat both as pre-flight checks before every HTTP request. In my experience, interviewers love probing this area because it separates candidates who have actually built scrapers from those reading a design doc for the first time.

Components added: