Webcrawler with Crawl4AI: a Small Python Project

Crawl4AI is a Python framework that opens a real browser, loads a page, and hands you clean Markdown back. That's handy when you need page content for search, RAG, reports, audits, or a quick internal tool and don't want to fight Playwright by hand.

I'm Andrew, and in this guide I'll build a small webcrawler with Crawl4AI. It does two things:

1. Starts from one URL and finds more pages with a breadth-first crawl.
2. Visits those pages again and saves Markdown plus JSON metadata for each one.

The full working project sits in tools/ next to this article. I tested the code with crawl4ai==0.9.1, so if a newer version changes the API, check the changelog before copying blindly.

What We're Building

Here's the whole shape of it, no more:

• input: one start URL
• config: max depth, max pages, output directory
• output: manifest.json, one Markdown file per page, one JSON metadata file per page

I keep discovery and processing as two separate steps. It's easier to debug that way.

First you ask: "which URLs did the crawler find?"

Then you ask: "what content did we pull from those URLs?"

That split matters the moment a crawl goes sideways. And crawls go sideways more often than you'd think.

Install Crawl4AI

Set up a virtual environment first:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

requirements.txt has one line:

crawl4ai==0.9.1

Crawl4AI runs on Playwright under the hood, so you also need the browser binaries:

crawl4ai-setup

If that doesn't pull in Chromium on your machine, run Playwright directly:

python -m playwright install --with-deps chromium

Check your local setup with:

crawl4ai-doctor

Minimal Single Page Crawl

Before you write a whole crawler, make sure one page works.

import asyncio
from crawl4ai import AsyncWebCrawler


async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown.raw_markdown)


asyncio.run(main())

That's it. It opens a browser, loads the page, pulls the content, and prints Markdown.

If this fails, fix the browser setup before writing more code. A crawler loop won't save you if Chromium can't even start.

Project Structure

Here's the small project:

tools/
  crawl.py
  requirements.txt
  README.md

Run it like this:

python crawl.py https://docs.crawl4ai.com \
  --max-depth 1 \
  --max-pages 10 \
  --out out

After a run, the output directory looks like this:

out/
  manifest.json
  pages/
    0000-index-a1b2c3d4e5.md
    0000-index-a1b2c3d4e5.json

The Markdown file is what you'll probably feed into another system. The JSON file keeps the crawl details: URL, status, links, metadata, errors.

Step 1: Configure the Browser

Crawl4AI exposes browser settings through BrowserConfig.

from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_config = BrowserConfig(
    headless=True,
    verbose=False,
)

async with AsyncWebCrawler(config=browser_config) as crawler:
    ...

For normal crawler runs, keep headless=True. If a page is misbehaving and you want to watch it load, run the sample with:

python crawl.py https://example.com --headed

That pops up a visible browser window so you can see what's actually happening.

Step 2: Discover Pages with BFS

For discovery, the sample uses BFSDeepCrawlStrategy.

BFS stands for breadth-first search. The crawler starts at the seed URL, grabs the links on that page, then moves outward one level at a time.

from crawl4ai import CacheMode, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

discovery_config = CrawlerRunConfig(
    prefetch=True,
    cache_mode=CacheMode.BYPASS,
    deep_crawl_strategy=BFSDeepCrawlStrategy(
        max_depth=1,
        max_pages=10,
        include_external=False,
    ),
)

results = await crawler.arun(start_url, config=discovery_config)

The settings that matter:

• max_depth: how far from the start URL to go
• max_pages: a hard page limit
• include_external: whether to follow links to other domains
• prefetch=True: fast, since discovery only needs the URL list

Keep include_external=False for most jobs. It's way too easy to start on one docs page and end up crawling half the internet through GitHub, Twitter, and CDN links you never meant to touch.

Step 3: Deduplicate URLs

Deep crawling can hand back the same URL more than once. The sample keeps the order stable while stripping duplicates:

seen: set[str] = set()
deduped_urls: list[str] = []

for url in discovered_urls:
    if url in seen:
        continue
    seen.add(url)
    deduped_urls.append(url)

This looks like boring code, and it is. But it saves you real time later. Duplicate pages mean duplicate files, duplicate embeddings, and reports that don't add up.

Step 4: Process Each Page

Discovery gives you the URLs. This second pass pulls the full Markdown and metadata.

full_config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    remove_overlay_elements=True,
)

for i, url in enumerate(deduped_urls[:max_pages]):
    result = await crawler.arun(url, config=full_config)

    md = ""
    if result.markdown is not None:
        md = result.markdown.raw_markdown or ""

remove_overlay_elements=True handles cookie banners, popups, and other page clutter that would otherwise leak into your extracted content.

It won't catch everything. Real websites are messy places. But it's a solid default for an example crawler.

Step 5: Save Markdown and JSON

The sample writes one Markdown file and one JSON file per page.

md_path.write_text(md, encoding="utf-8")

data = {
    "url": result.url,
    "success": bool(result.success),
    "status_code": result.status_code,
    "metadata": result.metadata,
    "links": result.links,
    "markdown": {
        "raw_markdown_len": len(md),
        "file": md_path.name,
    },
    "errors": result.error_message,
}

json_path.write_text(
    json.dumps(data, indent=2, ensure_ascii=True) + "\n",
    encoding="utf-8",
)

I save both formats, every time.

Markdown is what you read and feed into other text pipelines. JSON is what you dig through when something goes wrong.

Step 6: Write a Manifest

A manifest gives you one file that explains the whole run.

manifest = {
    "start_url": start_url,
    "duration_s": round(time.time() - started_at, 3),
    "config": {
        "max_depth": max_depth,
        "max_pages": max_pages,
        "include_external": include_external,
        "headless": headless,
    },
    "discovered": discovered,
    "processed": processed,
}

This pays off the second time you run the crawler. You can compare settings across runs and figure out fast whether a bad output came from your code, a site change, or a different crawl limit.

Full CLI

The full script takes a start URL and a handful of useful flags:

python crawl.py START_URL \
  --out out \
  --max-depth 1 \
  --max-pages 10

Other flags:

• --include-external: follow links to other domains
• --headed: show the browser window
• --verbose: print more Crawl4AI logs

Example:

python crawl.py https://docs.crawl4ai.com \
  --max-depth 1 \
  --max-pages 10 \
  --out out \
  --verbose

What This Crawler Handles Well

This crawler is a good fit for:

• small documentation sites
• product pages where JavaScript renders the content
• internal audits
• turning a handful of pages into Markdown
• testing Crawl4AI before you build something bigger

You get a real browser without hand-writing Playwright code, and Markdown comes out the other end, which is exactly what most AI and search workflows want.

What I'd Add for Production

I kept this sample small. For production I'd bolt on:

• robots.txt checks when crawling third-party sites
• retry logic for timeouts and one-off failures
• rate limits per domain
• proxy support for sites that block datacenter IPs
• persistent job state so a long crawl can pick up where it left off
• structured logs
• tests around file output and URL normalization

Crawl4AI gives you the browser and the extraction layer. You still own everything around it.

That tradeoff is fine for local tools and internal jobs. At real scale, the hard parts are scheduling, retries, proxy health, deduplication, storage, and monitoring, not the browser automation itself.

Final Code

The full code lives in tools/crawl.py. The main flow:

1. Create output directories.
2. Start AsyncWebCrawler.
3. Run BFS discovery.
4. Deduplicate discovered URLs.
5. Crawl each URL for Markdown.
6. Save page files and a manifest.

Start small:

python crawl.py https://example.com --max-depth 0 --max-pages 1 --out out

Then push the limits up:

python crawl.py https://docs.crawl4ai.com --max-depth 1 --max-pages 10 --out out

That's enough to see how the framework behaves before you turn a first run into a noisy, uncontrolled crawl.

Conclusion

A webcrawler built on Crawl4AI can stay small because the framework already handles browser automation and Markdown extraction for you.

The trick is keeping the crawler boring: discover URLs, dedupe them, process pages, save files you can actually inspect. Once that works, layer in retries, rate limits, proxies, queues, and storage based on the real site you're pointing it at.

For a small Python crawler, this is a practical place to start. If you end up needing thousands of pages, scheduled runs, or proxy rotation without babysitting the infrastructure yourself, that's the point where a hosted option like WebCrawlerAPI is worth a look.