What is robots.txt?

robots.txt is a plain-text file at a site's root that declares which paths crawlers may or may not request, per user agent. It's a convention, not a security boundary — nothing stops a crawler from ignoring it technically — but respecting it is the baseline expectation for any well-behaved crawler.

Common mistake

Crawling without checking it at all, or parsing it by hand with string matching:

def is_allowed(url, disallowed_paths):
    return not any(url.startswith(p) for p in disallowed_paths)  # ad-hoc, ignores user-agent scoping and wildcards

Hand-rolled parsing misses the actual rule syntax — user-agent-specific blocks, Allow overrides inside a broader Disallow, and wildcard patterns (Disallow: /search*) — so it either blocks paths that were actually allowed or crawls paths that weren't.

The fix

Use a proper robots.txt parser and check every URL against it before adding it to the crawl queue:

from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

class CrawlPolicy:
    def __init__(self, seed_url, user_agent="MyCrawler"):
        self.user_agent = user_agent
        parsed = urlparse(seed_url)
        self.rp = RobotFileParser()
        self.rp.set_url(f"{parsed.scheme}://{parsed.netloc}/robots.txt")
        self.rp.read()

    def can_fetch(self, url: str) -> bool:
        return self.rp.can_fetch(self.user_agent, url)

    def crawl_delay(self) -> float | None:
        return self.rp.crawl_delay(self.user_agent)

policy = CrawlPolicy("https://example.com")
if policy.can_fetch("https://example.com/blog/post-1"):
    fetch(url)

robots.txt also frequently points to a sitemap — read it from the same file to shortcut discovery instead of crawling link-by-link:

sitemap_urls = policy.rp.site_maps()  # returns declared sitemap URLs, if any

Why it works

RobotFileParser implements the actual specification — user-agent-scoped rules, Allow overrides within a Disallow block, wildcards — so can_fetch() returns the answer the site owner intended rather than an approximation from string prefix matching. Reading crawl_delay() from the same file respects any explicit pacing the site has requested, which matters more than a self-chosen delay when one is published. Checking site_maps() turns discovery from "guess which links matter" into "here's the URL list the site owner already published."

Tips

• Check robots.txt once per domain and cache the parsed result — re-fetching and re-parsing it per URL is wasted work and adds unnecessary requests to the crawl.
• robots.txt is not authentication or access control — a Disallow doesn't password-protect a path, it's a request that well-behaved crawlers honor voluntarily.
• If a sitemap is declared, prefer it over link-following for initial discovery — it's typically more complete and far cheaper than crawling the link graph.
• Ignoring robots.txt is one of the fastest ways to get blocked outright — see how to avoid getting blocked when crawling.

WebCrawlerAPI checks and respects robots.txt automatically on every crawl job, so compliance doesn't depend on you implementing the parser correctly yourself.