What just happened in the demo
A crawler started at one URL, pulled the page, found links and kept going, layer by layer, until it had mapped everything reachable from that starting point. That is the whole idea. Here is what happens under the hood.
The core loop: how a crawler thinks
The mechanics are simple: it is a loop. The hard parts come from everything around the loop, but the loop itself fits in your head.
Start with a seed URL
A crawl starts with one URL. It goes into a queue. Everything that follows is a consequence of what is on that first page.
Fetch the page
The crawler sends an HTTP GET request, the same as your browser when you type an address. The server sends back HTML. If the server blocks the request or returns an error, the crawler logs it and moves on.
Parse and extract links
The HTML is parsed to find every <a href> tag. Each href becomes a candidate URL. Internal links (same domain) go into the queue. External links are logged but skipped unless the crawl is configured to follow them.
Respect robots.txt and crawl delays
This step is not optional if you want to crawl responsibly. Before fetching pages from a domain, a real crawler checks the site's robots.txt. The file lists paths that are off-limits and how fast a crawler may make requests via the Crawl-delay directive. Sites watch for crawlers that ignore it and block them. A basic robots.txt looks like this:
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 2
User-agent: Googlebot
Allow: /The Disallow lines mark paths the crawler must skip. Crawl-delay: 2 means wait two seconds between requests.
Add new URLs to the queue
A discovered URL only goes into the queue if it has not been seen before. The crawler keeps a visited set, a record of every URL already fetched or queued. This prevents infinite loops. Without it you would circle the same pages forever.
Repeat
The loop runs until the queue is empty. At that point every page reachable from the seed URL, within whatever limits were set, has been visited.
Traversal strategy: BFS vs DFS
The queue controls the order pages are visited. That order is the traversal strategy, and it changes what you get out of a crawl.
Breadth-first search (BFS)
BFS visits all pages at the current depth before going deeper. Start at the homepage, follow all links on that page, then all links found on those pages, and so on. You get broad coverage fast. It is the default for most general-purpose crawlers.
Depth-first search (DFS)
DFS follows one branch as deep as it goes before backtracking. From the homepage it might go to a category page, then a product page, then a review, before coming back up. It helps when chasing a known deep path. The risk is spending all your time in one corner while the rest of the site sits untouched.
Why most crawlers choose BFS
If you hit a rate limit or must stop early, BFS means you have already seen the pages closest to the root, which are usually the most important. With DFS you might stop having seen only one deep corner of the site.
Types of web crawlers
The architecture looks similar across crawler types, but the goals and the rules they follow differ.
Search engine crawlers (Googlebot, Bingbot)
These index the web so search engines can return relevant results. They follow robots.txt and Crawl-delay closely, because their reputation and legal standing depend on it. Google publishes the IP ranges of Googlebot so sites can verify it.
AI training crawlers (Common Crawl, GPTBot)
Common Crawl is an open dataset used to train many large language models. OpenAI's GPTBot and Anthropic's ClaudeBot are crawlers built to collect training data. They have become a point of friction for site owners who want to opt out.
Site audit crawlers
Tools like Screaming Frog, Ahrefs and SEMrush crawl a site to find broken links, redirect chains, missing meta tags and thin content. They usually run on demand, not continuously.
Focused and niche crawlers
Price monitoring, news aggregation and real estate listings use focused crawlers that target one site or content type and go deep rather than broad. This is where WebCrawlerAPI is commonly used: teams that need to crawl a specific site repeatedly as part of a data pipeline or AI workflow.
When does a crawler stop?
The obvious answer is when the queue is empty. In production that rarely happens. Crawlers stop because a page limit was reached, a depth limit was hit, a time budget ran out, or rate limiting slowed the crawl to a halt. A crawler with no explicit stop conditions can run for days on a large site, so production crawlers always set bounds: max pages, max depth and max run time.
Web crawling vs web scraping
The terms get used interchangeably, but they differ. Crawling is discovery: following links across pages to map a site or collect URLs. Scraping is extraction: pulling specific data from a page you already found. A crawler finds pages; a scraper reads them. Most real pipelines do both.
| Web crawling | Web scraping | |
|---|---|---|
| Goal | Discover pages | Extract data |
| Input | Seed URL | Known page URL |
| Output | List of URLs / pages | Structured data fields |
| Scope | Site-wide | Page-level |
SEO and crawl budget: why this matters for your site
Search engine crawlers do not crawl every page of every site every day. They work within a crawl budget, a limit on how many pages they fetch from your site per cycle. If your site has thousands of URLs, low-value pages such as thin content, duplicate URLs, infinite pagination and session-token URLs use up that budget before Googlebot reaches your important pages. The fix is a clean URL structure, a maintained sitemap and a robots.txt that blocks pages you do not need indexed.
The hard parts
The core loop is simple. Everything around it is not.
JavaScript rendering
Many modern sites load content with JavaScript after the initial HTML arrives. A basic HTTP crawler parses the HTML, but the content was never in it. You need a headless browser such as Puppeteer or Playwright that runs the scripts and waits for rendering to finish. See the JavaScript rendering guide.
Duplicate and infinite URLs
The same page can appear at /about, /about/, /About and /about?ref=nav. Without URL normalization all four get crawled separately. Some sites generate effectively infinite URLs: calendars that link forward forever, session tokens in every URL, pagination with no end. A production crawler has to detect and break these loops.
Rate limiting and politeness at scale
Crawl too fast and you get 429s, IP bans, or you overload a server. Respect Crawl-delay, add your own delays and back off exponentially on rate-limit responses. At scale this also means spreading requests over time.
Anti-bot measures
Cloudflare, Datadome and PerimeterX identify non-human traffic from headers, TLS handshake patterns, IP reputation and behavior. A basic crawler is blocked quickly on a well-defended site. Getting through takes rotating proxies, realistic browser fingerprints and ongoing maintenance as detection rules change.
Web crawling for AI workflows
Most of the recent growth in crawling is driven by AI use cases. LLMs need clean text at scale, and crawling is the main way to get it: teams crawl documentation sites, help centers or whole domains to build a knowledge base the model queries at runtime. Output format matters here. Raw HTML is full of navigation, ads and scripts that pollute the context window, so you want clean markdown or plain text. WebCrawlerAPI returns markdown for this reason.
Build vs buy: when to use a crawler API
Building a basic crawler takes an afternoon. Building one that handles JavaScript rendering, anti-bot defenses, retries, deduplication, proxy rotation, robots.txt compliance and stable output is weeks of engineering plus ongoing maintenance. If crawling is the product you sell, build it. If it is infrastructure for something else, such as a knowledge base, a data product or an SEO tool, an API that already handles it is almost always faster and cheaper. Compare options in best web crawler APIs, or follow how to build a web crawler yourself.