What is web crawling?

Web crawling is discovery: starting from one or a few seed URLs, following links to find every other page on a site, and keeping track of what's already been visited so you don't loop forever. It answers "what pages exist," which is a prerequisite question before you can extract anything from them.

Common mistake

Following every link with no visited-set, no depth limit, and no domain boundary:

def crawl(url, urls=None):
    urls = urls or []
    urls.append(url)
    html = requests.get(url).text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select("a[href]"):
        crawl(urljoin(url, a["href"]), urls)  # no visited check, no domain check
    return urls

Without a visited set, two pages linking to each other recurse infinitely. Without a domain check, the crawler walks off-site the first time it hits an external link, and can end up trying to crawl the entire web from one seed URL.

The fix

Track visited URLs explicitly, bound the crawl to the target domain, and use a queue instead of recursion so it doesn't blow the call stack on a deep site:

from collections import deque
from urllib.parse import urljoin, urlparse

def crawl(seed_url, max_pages=1000):
    domain = urlparse(seed_url).netloc
    visited = set()
    queue = deque([seed_url])
    pages = []

    while queue and len(pages) < max_pages:
        url = queue.popleft()
        if url in visited:
            continue
        visited.add(url)

        resp = requests.get(url, timeout=10)
        pages.append({"url": url, "status": resp.status_code})

        if "text/html" not in resp.headers.get("Content-Type", ""):
            continue

        soup = BeautifulSoup(resp.text, "html.parser")
        for a in soup.select("a[href]"):
            next_url = urljoin(url, a["href"]).split("#")[0]  # strip fragments
            if urlparse(next_url).netloc == domain and next_url not in visited:
                queue.append(next_url)

    return pages

Why it works

The visited set guarantees each URL is fetched once regardless of how many pages link to it, which is what actually stops infinite loops — not depth or count limits, which only mask the underlying cycle. Bounding by domain keeps the crawl scoped to the site you're actually interested in instead of following the web's link graph outward indefinitely. A queue (BFS) instead of recursion (DFS) avoids Python's recursion limit on deep sites and processes pages roughly in order of proximity to the seed, which tends to surface the most important pages first.

Tips

• Strip URL fragments (#section) before adding to the visited set — /page#top and /page#bottom are the same page and will otherwise be crawled twice.
• max_pages is a safety cap, not a strategy — see what crawl budget is for how to allocate it deliberately rather than just capping it.
• Check robots.txt before adding URLs to the queue, not after fetching them — see what robots.txt does.
• Crawling only gets you URLs and raw content — extracting specific fields from each page is a separate step; see how crawling differs from scraping.

WebCrawlerAPI runs this discovery step for you — point it at a domain and it returns the full page list with rendered content, handling visited-tracking, robots.txt, and rate limiting internally.