Extract clean markdown, HTML, and metadata from any website with one pip install. Works seamlessly with Django, Flask, FastAPI, and Jupyter notebooks.
Looking for other integrations? Visit main landing page
Native async support for Django, Flask, FastAPI, and LangChain. Works in Jupyter notebooks and data science workflows.
Get clean markdown or HTML perfect for RAG pipelines, embeddings, and AI training data. No HTML parsing needed.
Skip the complexity of managing proxies, browsers, and anti-bot systems. We handle infrastructure so you focus on your app.
# pip install webcrawlerapi
from webcrawlerapi import WebCrawlerAPI
crawler = WebCrawlerAPI(api_key="YOUR_API_KEY")
response = crawler.crawl(
url="https://example.com",
output_formats=["markdown"],
items_limit=10
)
print(response.status, response.items[0].markdown)# For FastAPI, Django Async, etc.
from webcrawlerapi import AsyncWebCrawlerAPI
import asyncio
async def main():
crawler = AsyncWebCrawlerAPI(api_key="YOUR_API_KEY")
response = await crawler.crawl(
url="https://example.com/docs",
output_formats=["markdown"],
items_limit=50
)
print(f"Crawled {len(response.items)} pages")
asyncio.run(main())