LangChain Integration

Seamlessly convert websites and webpages into markdown format for LLM data processing pipelines

The WebCrawlerAPI LangChain integration allows you to seamlessly convert websites and webpages into markdown or cleaned content format, making it perfect for LLM data processing pipelines. This integration requires no subscription and provides a straightforward way to incorporate web crawling capabilities into your LangChain document processing workflow.

Installation

First, obtain your API key from WebCrawlerAPI, then install the package using pip. Requires Python 3.10 or newer.

bash

pip install webcrawlerapi-langchain

The loader reads your key from the WEBCRAWLERAPI_API_KEY environment variable when api_key is not passed. Likewise, WEBCRAWLERAPI_BASE_URL is used for base_url when it is not set.

Usage

Basic Loading

The simplest way to use the WebCrawlerAPI loader is through the basic loading method:

python

from webcrawlerapi_langchain import WebCrawlerAPILoader

# Initialize the loader
loader = WebCrawlerAPILoader(
    url="https://example.com",
    api_key="your-api-key",
    output_format="markdown",
    items_limit=10
)

# Load documents
documents = loader.load()

# Use documents in your LangChain pipeline
for doc in documents:
    print(doc.page_content[:100])
    print(doc.metadata)

Each Document contains the page content in the selected output_format and the following metadata keys: url, title, status_code, created_at, referred_url, depth, cost, job_id, and item_id.

Advanced Loading Methods

The SDK supports multiple loading patterns to suit different use cases:

Async Loading

For asynchronous operations:

python

# Async loading
documents = await loader.aload()

Lazy Loading

When dealing with large datasets. Each page is yielded as soon as it finishes crawling, without waiting for the whole job:

python

# Lazy loading
for doc in loader.lazy_load():
    print(doc.page_content[:100])

Async Lazy Loading

Combining asynchronous and lazy loading:

python

# Async lazy loading
async for doc in loader.alazy_load():
    print(doc.page_content[:100])

Configuration Options

The WebCrawlerAPILoader accepts the following configuration parameters:

Parameter	Type	Description
`url`	string	The target URL to crawl
`api_key`	string	Your WebCrawlerAPI API key. Defaults to the `WEBCRAWLERAPI_API_KEY` environment variable
`base_url`	string	API base URL. Defaults to the `WEBCRAWLERAPI_BASE_URL` environment variable
`output_format`	string	Content format of each Document: `"markdown"` (default), `"cleaned"`, or `"html"`
`items_limit`	integer	Maximum number of pages to crawl. Default `10`
`whitelist_regexp`	string	Regex pattern for URL whitelist
`blacklist_regexp`	string	Regex pattern for URL blacklist
`main_content_only`	boolean	Extract only the main content of each page. Default `false`
`max_depth`	integer	Maximum crawl depth. `0` crawls the seed URL only, `1` crawls the seed URL and its direct links
`max_age`	integer	Maximum age of cached content in seconds. `0` always fetches fresh content
`respect_robots_txt`	boolean	Whether to respect robots.txt. Default `false`
`keep_query_params`	boolean	Keep URL query parameters when deduplicating links
`max_polls`	integer	Maximum number of job status checks before giving up. Default `100`

Best Practices

Rate Limiting: Be mindful of API rate limits when crawling multiple pages.
Error Handling: Always implement proper error handling for network issues and API responses.
Content Type: Choose the appropriate output_format based on your LLM's requirements:
- Use "markdown" for structured content
- Use "cleaned" for plain text
- Use "html" for raw HTML content

Example Use Cases

Knowledge Base Search

You can find working code example in the GitHub Book Information Extractor with LangChain and WebcrawlerAPI

Crawl a docs site, split pages on markdown headings, and index the chunks in a vector store. Requires langchain-openai and langchain-text-splitters. InMemoryVectorStore and OpenAI embeddings are examples; swap in your own.

python

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import MarkdownHeaderTextSplitter
from webcrawlerapi_langchain import WebCrawlerAPILoader

# Load documents from a website
loader = WebCrawlerAPILoader(
    url="https://docs.example.com",
    output_format="markdown",
    main_content_only=True,
    items_limit=50
)
documents = loader.load()

# Split each page on headings, keeping the source URL on every chunk
splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2")],
    strip_headers=False
)
chunks = []
for doc in documents:
    for chunk in splitter.split_text(doc.page_content):
        chunk.metadata.update(url=doc.metadata["url"], title=doc.metadata["title"])
        chunks.append(chunk)

# Index and search
store = InMemoryVectorStore(OpenAIEmbeddings(model="text-embedding-3-small"))
store.add_documents(chunks)

for hit in store.similarity_search("How do I get started?", k=3):
    print(hit.metadata["url"], hit.page_content[:100])

Error Handling

Invalid parameters raise ValueError. Crawling and API failures raise WebCrawlerAPILoaderError, importable from webcrawlerapi_langchain. Its message includes the failed item URLs and error codes when a job has failed items:

python

from webcrawlerapi_langchain import WebCrawlerAPILoader, WebCrawlerAPILoaderError

try:
    loader = WebCrawlerAPILoader(
        url="https://example.com",
        api_key="your-api-key"
    )
    documents = loader.load()
except WebCrawlerAPILoaderError as e:
    print(f"Crawl Error: {e}")
except ValueError as e:
    print(f"Configuration Error: {e}")

Support

For additional support or to report issues, please visit the WebCrawlerAPI documentation or the GitHub repository.