docs
SDKs and Code Examples
MCP (Model Context Protocol)

MCP WebcrawlerAPI integration

MCP (Model Context Protocol) is a protocol that allows AI assistants to connect to external services and tools. The WebCrawlerAPI MCP server enables AI assistants like Claude Code to scrape web content directly during conversations.

Installation

Using npm

npm install -g webcrawler-mcp

Using npx (no installation required)

npx webcrawler-mcp

Setup

1. Get your API key

Get your API key from the WebCrawlerAPI dashboard (opens in a new tab).

2. Set environment variable

export WEBCRAWLER_API_KEY="your-api-key-here"

3. Run the server

The MCP server can run in two modes:

Standard mode (stdio):

npx webcrawler-mcp

HTTP mode:

export USE_HTTP=true
export PORT=8080
npx webcrawler-mcp

Configuration

The MCP server supports the following environment variables:

  • WEBCRAWLER_API_KEY (required): Your WebCrawlerAPI access key
  • USE_HTTP (optional): Enable HTTP transport mode
  • PORT (optional): HTTP server port (default: 8080)

Using with Claude Code

Once the MCP server is running, Claude Code can automatically discover and use the webcrawler-scrape tool to scrape web content.

The tool accepts these parameters:

  • url (required): The webpage URL to scrape
  • prompt (optional): Specific information to extract from the page

Example usage in Claude Code:

Please use webcrawler-scrape to scrape https://example.com and extract the main article content

Claude Code will automatically use the MCP server to scrape the webpage and return the content as markdown.

Features

  • Single webpage scraping: Extract content from any accessible webpage
  • Content extraction: Use prompts to extract specific information
  • Markdown output: Clean, structured content ready for AI processing
  • Real-time scraping: Dynamic content extraction during AI conversations

Development

For development and testing:

git clone https://github.com/WebCrawlerAPI/webcrawlerapi-mcp
cd webcrawlerapi-mcp
npm install
npm run dev

Build for production:

npm run build
npm start

GitHub Repository

WebCrawlerAPI MCP Server on GitHub (opens in a new tab)

If you have any questions, please contact us at [email protected]

🦜🔗 LangChainn8n