Give your OpenClaw agent clean markdown
WebCrawler API turns any URL into clean markdown your OpenClaw skills can act on — instantly. Scrape a page or crawl an entire site. Proxies, JS rendering, and CAPTCHAs handled for you.
$ openclaw skills install webcrawlerBuilt for AI agents
Everything your Claw skill needs to read the web
One API. Any URL. Clean data. Ship your web research skill in minutes.
Markdown extraction
Clean markdown, ready for your skill
We load the page, strip menus, cookie banners, ads, and navigation, then return structured markdown. Paste it directly into your Claw skill handler or store it in your agent's memory — no extra cleanup step.
- Noise removed for you. Menus, footers, ads, cookie notices stripped before the response leaves our servers.
- Works with any site. JS-rendered SPAs, paywalled previews, or plain HTML — we handle rendering with headless Chromium.
Infrastructure
No scraping infra to manage
Proxies, retries, headless browsers, CAPTCHAs, and anti-bot protection are all handled by our stack. Your Claw skill stays simple — one API call, clean response.
- Global proxy network. Residential and datacenter proxies rotated automatically per request.
- Smart routing. Each request sent through the fastest path that can actually get the page.
Frequently Asked Questions
Common questions from OpenClaw skill builders
How do I give my OpenClaw skill access to a website?
Run openclaw skills install webcrawler to add the skill, then get your API key from the dashboard. After that your Claw agent can scrape any URL and get back clean markdown — no DOM parsing, no browser automation to set up.
Can my Claw agent crawl an entire docs site, not just one page?
Yes. Use the /crawl endpoint with a starting URL and optional depth limit. We discover and fetch all linked pages, returning clean markdown for each. Ideal for indexing documentation into your agent's memory.
Does it handle JavaScript-heavy sites?
Yes. We use headless Chromium to render pages before extraction, so JS-rendered content, SPAs, and dynamic dashboards all work. You don't need to manage a browser instance in your skill.
Do I need to pay a subscription?
No subscription required. Pay-as-you-go starts at $0.002 per page — you only pay for successful requests. Subscription plans are available for higher volumes at lower per-page rates.
Can I try it before adding it to my Claw skill?
Yes — sign up at dash.webcrawlerapi.com, no credit card required.
