Turn docs and websites into clean markdown for AI agents
Point it at any docs site, help center, or website. Get back clean markdown, ready to index into your AI agent.
Try it free — no credit card required
Trusted by 1,000+ developers at
Powering AI support bots and knowledge products
Integrate in 60 seconds
Works with every major language. No boilerplate. No friction.
curl -X POST https://api.webcrawlerapi.com/v1/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.stripe.com",
"items_limit": 10
}'Built for production
Built for AI teams who need reliable data
Smart defaults and powerful controls, ready to use out of the box.
Markdown extraction
Extract clean markdown from any page
We load the page, extract the markdown, clean it, remove clutter and junk, and return only the useful content. The result is formatted markdown that is ready to use in your AI agent.
- Cleaned and formatted for you.
- We remove menus, cookie banners, footers, ads, and other junk, then format the markdown so it is easy to work with.
- Ready for your AI agent.
- Use the result directly in prompts, indexing, or storage without adding another cleanup step.
# Getting Started - OpenClaw ## [](#getting-started)Getting Started Install OpenClaw, run onboarding, and chat with your AI assistant — all in about 5 minutes. By the end you will have a running Gateway, configured auth, and a working chat session. ## [](#what-you-need)What you need * **Node.js** — Node 24 recommended (Node 22.16+ also supported) * **An API key** from a model provider (Anthropic, OpenAI, Google, etc.) — onboarding will prompt you Check your Node version with `node --version`. **Windows users:** both native Windows and WSL2 are supported. WSL2 is more stable and recommended for the full experience. See [Windows](https://docs.openclaw.ai/platforms/windows). Need to install Node? See [Node setup](https://docs.openclaw.ai/install/node). ## [](#quick-setup)Quick setup 1 [](#) Install OpenClaw * macOS / Linux * Windows (PowerShell) ``` curl -fsSL https://openclaw.ai/install.sh | bash ``` ``` iwr -useb https://openclaw.ai/install.ps1 | iex ``` Other install methods (Docker, Nix, npm): [Install](https://docs.openclaw.ai/install). 2 [](#) Run onboarding ``` openclaw onboard --install-daemon ``` The wizard walks you through choosing a model provider, setting an API key, and configuring the Gateway. It takes about 2 minutes.See [Onboarding (CLI)](https://docs.openclaw.ai/start/wizard) for the full reference. 3 [](#) Verify the Gateway is running ``` openclaw gateway status ``` You should see the Gateway listening on port 18789. 4 [](#) Open the dashboard ``` openclaw dashboard ``` This opens the Control UI in your browser. If it loads, everything is working. 5 [](#) Send your first message Type a message in the Control UI chat and you should get an AI reply.Want to chat from your phone instead? The fastest channel to set up is [Telegram](https://docs.openclaw.ai/channels/telegram) (just a bot token). See [Channels](https://docs.openclaw.ai/channels) for all options. ## [](#what-to-do-next)What to do next ## Connect a channel WhatsApp, Telegram, Discord, iMessage, and more. ## Pairing and safety Control who can message your agent. ## Configure the Gateway Models, tools, sandbox, and advanced settings. ## Browse tools Browser, exec, web search, skills, and plugins. Advanced: environment variables If you run OpenClaw as a service account or want custom paths: * `OPENCLAW_HOME` — home directory for internal path resolution * `OPENCLAW_STATE_DIR` — override the state directory * `OPENCLAW_CONFIG_PATH` — override the config file path Full reference: [Environment variables](https://docs.openclaw.ai/help/environment).
Smart caching
Cached pages in under a second
Frequently requested pages are returned from smart cache in about 0.9 seconds instead of 9 seconds. When you need to bypass cache, pass max_age=0.
max_age=0 to skip cacheChange detection
Get only what changed
Set up a feed for any site and receive only the pages that changed, along with the updated content. No polling loops. No duplicate fetches. No wasted tokens.
Explore FeedsInfrastructure
Stop managing scraping infra
Proxies, retries, headless browsers, CAPTCHAs, anti-bot protection, JavaScript rendering. We handle the stack and route each request through the fastest path that can actually get the page.
By the numbers
Reliable infrastructure, proven in production.
Real usage, fast runs, stable uptime, and extraction performance in one control panel.
Active builders
Teams building in production
Extraction quality
Success rate
Fast turnaround
Average crawling time
Platform stability
Uptime
Without writing a line of code
No-code integrations
Use WebCrawlerAPI with the no-code tools your team already knows. Pick a platform, open the guide, and connect crawling to your workflow.
Zapier
Connect crawls to thousands of apps with quick trigger-based automations.
Make
Build flexible no-code scenarios for scraping, enrichment, and delivery.
n8n
Run automations your way with a visual flow tool that stays developer-friendly.
Integrately
Launch simple automations quickly with prebuilt no-code connection patterns.
Pricing
Simple, transparent pricing
Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.
Pay As You Go
No commitment
From $0.002 / page
- Unlimited proxy included
- Up to 5 parallel requests
- Pay only for successful requests
- Content cleaning included
- Run prompts over content for extra 0.002$
Standard
Best for growing teams
$99/month
- From $0.0015 / page
- Unlimited proxy included
- Up to 50 parallel requests
- Pay only for successful requests
- Content cleaning included
- Run prompts over content for extra 0.002$
Scale
For high-volume crawling
$499/month
- From $0.001 / page
- Unlimited proxy included
- Up to 50 parallel requests
- Pay only for successful requests
- Content cleaning included
- Run prompts over content for extra 0.002$
Frequently Asked Questions
Everything you need to know about our web crawling service