Keep track of updates and improvements to our platform
Cache hits are now free across WebCrawlerAPI.
When a scrape request matches a previously fetched page, the cached result is returned at no cost.
When a run matches the same model, prompt, and URLs as a prior run, the cached result is returned at no cost.
Meet the Crawling Agent — an AI-powered service that browses websites, follows links, and returns structured JSON, all from a natural-language prompt.
Unlike standard scraping that fetches a single URL, the agent acts like a researcher: it can perform web searches, read pages, decide which links are relevant, follow them, and extract the data you asked for. It autonomously decides how to find and extract the requested information. You describe what you want; the agent figures out how to get it.
You pay per LLM token used and per page scraped during a run. A typical task can cost anywhere from a fraction of a cent up to the max_spend_usd limit you set, depending on complexity. All spending is visible in your dashboard.
Check out the Crawling Agent documentation for prompt-writing tips, examples, and response details, or jump straight to the API reference.
A set of quality-of-life improvements landed this month across the API and dashboard.
Crawl jobs now automatically fetch and parse sitemap.xml when starting a new crawl. Pages found in the sitemap are added to the initial link queue, so the crawler reaches more pages earlier - without any changes to your requests.
Each job in the jobs list now shows how much it cost. Useful for tracking spend per crawl or scrape run without having to cross-reference billing statements.
You can now filter your jobs list by status — done, processing, failed, and others. Quickly find the runs you care about without scrolling through everything.
The synchronous /v2/scrape endpoint now waits up to 3 minutes (previously 2 minutes) before timing out. Heavy pages and slower sites are less likely to return a timeout error.
Top-up payments now go through Stripe Checkout, which automatically calculates and applies taxes based on your billing region.
The /v2/scrape endpoint now accepts an output_formats array, letting you request multiple output representations in a single API call.
{
"url": "https://example.com",
"output_formats": ["markdown", "links"]
}
The response will include both the markdown content and a links array with all URLs found on the page.
See the scrape documentation for the full list of parameters.
A new endpoint GET /organization/balance is now available to retrieve your organization's current spendable balance programmatically.
curl --request GET \
--url 'https://api.webcrawlerapi.com/organization/balance' \
--header 'Authorization: Bearer YOUR_API_KEY'
See the documentation for full details.
You can now create and manage multiple API keys per organization from the Access Keys page in the dashboard.
This makes it easier to rotate credentials, isolate environments, and revoke access for a specific integration without disrupting others.
This March we shipped three customer-facing improvements across billing and extraction workflows.
You can now switch to a monthly subscription directly from the dashboard and manage your plan without contacting support.
This includes:
Prompt-based scraping on /v2/scrape now supports structured outputs.
When you send a prompt, you can also provide a response_schema so the response follows a JSON Schema-defined structure.
Useful for:
Completed markdown crawl jobs can now be exported as a single combined markdown file.
Instead of fetching each page result one by one, you can download one consolidated output for the full job.
Useful for:
Structured Outputs ensure AI-generated responses adhere to a JSON schema you define, eliminating the need to validate or retry incorrectly formatted responses.
When you provide a prompt to the /v2/scrape endpoint, you can now add a response_schema parameter to enforce strict JSON Schema validation. The API will return guaranteed structured data that matches your schema every time.
Perfect for:
Structured outputs cost the same as regular prompts: $0.002 per request (in addition to the base crawling cost).
See the Structured Outputs documentation for schema examples and complete usage guide.
New GET /job/:id/markdown endpoint returns the full job content in one single concatenated markdown file.
Instead of downloading individual page results, you can now get all crawled markdown content combined into a single file. Each page's content is separated with URL headers for easy parsing.
Perfect for:
See the API documentation for usage examples and complete reference.
Turn any website into a feed. Monitor websites for changes and get automatic updates via RSS, JSON feeds, or webhooks.
The new Feeds API lets you track content changes on any website without building custom monitoring infrastructure. Perfect for tracking blogs, news sites, documentation, or any web content that doesn't offer native feeds.
Create a feed with a single API call, then subscribe to updates via RSS/Atom, JSON Feed, or webhooks. The system automatically crawls your target website periodically and delivers only what changed.
See the Feeds documentation and API Reference for complete details.
WebcrawlerAPI is now available as an MCP (Model Context Protocol) server.
You can now integrate web scraping directly into AI workflows through MCP-compatible applications like Claude Code. The MCP server provides a webcrawler-scrape tool for extracting webpage content.
The MCP server includes:
The integration allows AI assistants to dynamically scrape and process web content during conversations, enabling real-time data analysis and content extraction workflows.
Read How to use WebcrawlerAPI with MCP
Connect your WebcrawlerAPI account and enhance your AI workflows with web scraping capabilities.
WebcrawlerAPI is now available on n8n.
You can now automate web scraping workflows with other apps through n8n. The integration uses the Scrape API v2 endpoint to scrape single webpages.
The n8n node includes actions for:
The integration supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.
Connect your WebcrawlerAPI account and start building automated workflows.
WebcrawlerAPI is now available on Make (formerly Integromat).
You can now automate web scraping workflows with other apps through Make. The integration uses the Scrape API v2 endpoint to scrape single webpages.
The Make app includes for now a single Scrape a single webpage action. It supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.
To use the WebcrawlerAPI in the Make app just search for "WebcrawlerAPI" in the app store.
Read our documentation on how to integrate WebcrawlerAPI in Make.
You can now automate web scraping workflows with 1200+ apps through Integrately. The integration provides seamless connectivity with popular business tools and platforms.
The Integrately app includes actions for:
The integration supports all WebcrawlerAPI features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.
Connect your WebcrawlerAPI account and start building automated workflows with Integrately's powerful automation platform.
WebcrawlerAPI is now available on Zapier.
You can now automate web scraping workflows with other apps through Zapier. The integration uses the Scrape API v2 endpoint to scrape single webpages.
The Zapier app includes actions for:
The integration supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.
Read How to integrate no-code webcrawler in Zapier
Connect your WebcrawlerAPI account and start building automated workflows.
Scrape API v2 is now live.
Easier and straight forward. The new version lets you run a prompt on the page. You can get results in markdown, cleaned text, or HTML. Scraping is now in synchronous mode, with a single API call.
The new endpoint is at https://api.webcrawlerapi.com/v2/scrape. See the API Reference for details.
Postman collection is here to help you with WebcrawlerAPI testing.
Methods included in the collection:
Get your API key and try it out.
WebcrawlerAPI now supports direct export to any S3 compatible storage.
When starting a job via API, just add several parameters, like access_key_id, secret_access_key and a few others. Crawled data will be placed under the specified path. Your keys will be deleted after the job ends. Read Upload to S3 docs for detailed information.
Organizations has been added to WebcrawlerAPI. This feature lets multiple team members use the same API account with different access levels.
To add team members, go to your dashboard and click the "Invite member" button. You can assign roles based on what each person needs to do. This lets developers use the API without seeing billing details or changing the team.
We're thrilled to announce the release of our official LangChain integration! The new webcrawlerapi-langchain package makes it seamless to incorporate WebcrawlerAPI's powerful web crawling capabilities into your LangChain document processing pipelines.
pip install webcrawlerapi-langchain
from webcrawlerapi_langchain import WebCrawlerAPILoader
loader = WebCrawlerAPILoader(
url="https://example.com",
api_key="your-api-key",
output_formats=["markdown"]
)
documents = loader.load()
Check our WebcrawlerAPI examples
Check out our LangChain SDK documentation for detailed usage instructions and examples. Start building powerful AI applications with web data today!
We're excited to announce that all new WebcrawlerAPI accounts now receive a $10 evaluation balance for a 7-day trial period! This initiative allows new users to thoroughly test our API capabilities without any upfront commitment.
The new trial balance makes it easier than ever to evaluate WebcrawlerAPI and test its capabilities for your projects.
Major dashboard improvements 💫
Major Update 🚀
Launched free llmstxt Generator Tool that helps create standardized llms.txt files for documenting AI models in your projects. You can learn more about the llms.txt standard in our detailed guide.
Major WebcrawlerAPI update: Comprehensive error handling system implementation
Major improvements to our headless browser implementation for enhanced web scraping capabilities:
The issue lasted for 9 hours but was not related to crawling. The root cause was a network issue affecting the monitoring server. Because the monitoring server was unavailable to the main job manager, each job report had to wait several minutes for a timeout response from the monitoring server.
As a result, the processing time for each job increased, and the job queue grew to several thousand jobs.
The incident has now been resolved. We are continuously working on improving our monitoring system to prevent similar issues in the future.
A status page link has been added to the website footer. The current status of WebCrawlerAPI services can now be checked at status.webcrawlerapi.com.
A changelog page has been added to the website. This page tracks all the changes, improvements, and fixes to WebCrawlerAPI.
A new tool Webpage to Markdown has been added. This tool converts any documentation or website into a beautiful Markdown file. It is free and does not require an API key. It can crawl up to 100 pages.
PDF content rendering has been implemented. Text content can now be extracted from PDF files. When a website contains a PDF file, its content will be extracted and returned in the response as page content.