WebcrawlerAPI product updates

Structured Outputs ensure AI-generated responses adhere to a JSON schema you define, eliminating the need to validate or retry incorrectly formatted responses.

When you provide a prompt to the /v2/scrape endpoint, you can now add a response_schema parameter to enforce strict JSON Schema validation. The API will return guaranteed structured data that matches your schema every time.

Perfect for:

• Data Extraction: Extract product info, business details, or any structured data with type safety
• Reliable Parsing: No more validating or handling malformed JSON responses
• Complex Structures: Support for nested objects, arrays, enums, and optional fields
• API Integration: Feed predictable data directly into your application or database

What's new?

• JSON Schema Validation: Define exact structure using standard JSON Schema format
• Type Safety: Guaranteed response format with string, number, boolean, object, array, and enum types
• Nested Structures: Support for complex nested objects and arrays of objects
• Enum Constraints: Restrict values to predefined options
• Optional Fields: Use null union types for fields that may not always be present
• Clear Error Handling: Validation errors returned immediately if schema is invalid

Pricing

Structured outputs cost the same as regular prompts: $0.002 per request (in addition to the base crawling cost).

See the Structured Outputs documentation for schema examples and complete usage guide.

New GET /job/:id/markdown endpoint returns the full job content in one single concatenated markdown file.

Instead of downloading individual page results, you can now get all crawled markdown content combined into a single file. Each page's content is separated with URL headers for easy parsing.

Perfect for:

• RAG Applications: Feed combined documentation into vector databases or AI models
• Batch Processing: Process entire website content at once for analysis or indexing
• Documentation Extraction: Extract and combine docs from multiple pages
• Backup: Archive complete crawl results in a single file

See the API documentation for usage examples and complete reference.

Turn any website into a feed. Monitor websites for changes and get automatic updates via RSS, JSON feeds, or webhooks.

The new Feeds API lets you track content changes on any website without building custom monitoring infrastructure. Perfect for tracking blogs, news sites, documentation, or any web content that doesn't offer native feeds.

What's new?

• RSS/Atom Feeds: Subscribe to any website in standard Atom 1.0 format compatible with all feed readers
• JSON Feed Format: Get updates in JSON Feed format for easy integration with applications
• Webhook Notifications: Receive instant POST requests when content changes are detected
• Automatic Monitoring: Periodic crawling with smart change detection
• Flexible Configuration: Control crawl depth, page limits, URL patterns, and output formats (markdown, cleaned, HTML)
• Error Resilience: Automatic pause after 3 consecutive errors to prevent unnecessary charges
• Status Tracking: Detailed metrics on pages crawled, changed, new, unavailable, and errors

How it works

Create a feed with a single API call, then subscribe to updates via RSS/Atom, JSON Feed, or webhooks. The system automatically crawls your target website periodically and delivers only what changed.

See the Feeds documentation and API Reference for complete details.

• Max depth parameter: New max_depth parameter for crawling to control link following depth
• Usage endpoint: Admin endpoint to retrieve usage statistics and metrics with date range filtering and optional daily breakdown (documentation)
• Microsoft login fix: Fixed authentication issues with Microsoft/Outlook email accounts
• Documentation redesign: New documentation design with copy-to-markdown feature for easy sharing with AI assistants
• PDF parsing improvements: Enhanced PDF content extraction and parsing
• CSV/TSV file parsing: Added support for parsing CSV and TSV files directly from URLs

• Google News feed scraping: Fixed and now working reliably
• New Integration Page: Added to dashboard showing all available WebCrawlerAPI integrations including code, no-code, and storage options
• Webhook status tracking: Now visible in dashboard with "Resend" button to retry failed webhook deliveries
• Infrastructure optimizations: Enhanced crawling and scraping performance
• WWW handling: Websites with and without www subdomain are now processed correctly
• Model upgrade: Switched to google/gemini-2.5-flash-lite for prompt processing - significantly faster than OpenAI models
• Main content extraction: New parameter to extract only useful text from webpages, perfect for blog posts and articles
• Milestone achieved: WebCrawlerAPI crossed 750K total crawled pages this month

WebcrawlerAPI is now available as an MCP (Model Context Protocol) server.

You can now integrate web scraping directly into AI workflows through MCP-compatible applications like Claude Code. The MCP server provides a webcrawler-scrape tool for extracting webpage content.

The MCP server includes:

• Single webpage scraping with markdown output
• Content extraction with optional prompts
• Seamless integration with Claude Code and other MCP clients

The integration allows AI assistants to dynamically scrape and process web content during conversations, enabling real-time data analysis and content extraction workflows.

Read How to use WebcrawlerAPI with MCP

Connect your WebcrawlerAPI account and enhance your AI workflows with web scraping capabilities.

WebcrawlerAPI MCP Server on GitHub

WebcrawlerAPI is now available on n8n.

You can now automate web scraping workflows with other apps through n8n. The integration uses the Scrape API v2 endpoint to scrape single webpages.

The n8n node includes actions for:

• Single webpage scraping
• Content extraction with prompts
• Multiple output formats (markdown, cleaned text, HTML)
• CSS selector-based content cleaning
• Custom headers and parameters

The integration supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.

Connect your WebcrawlerAPI account and start building automated workflows.

Install the n8n-nodes-webcrawlerapi package

WebcrawlerAPI is now available on Make (formerly Integromat).

You can now automate web scraping workflows with other apps through Make. The integration uses the Scrape API v2 endpoint to scrape single webpages.

The Make app includes for now a single Scrape a single webpage action. It supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.

To use the WebcrawlerAPI in the Make app just search for "WebcrawlerAPI" in the app store.

WebcrawlerAPI on Make

Read our documentation on how to integrate WebcrawlerAPI in Make.

You can now automate web scraping workflows with 1200+ apps through Integrately. The integration provides seamless connectivity with popular business tools and platforms.

The Integrately app includes actions for:

• Scrape a single webpage
• Start crawling job
• Get crawling job result

The integration supports all WebcrawlerAPI features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.

Connect your WebcrawlerAPI account and start building automated workflows with Integrately's powerful automation platform.

WebcrawlerAPI on Integrately

WebcrawlerAPI is now available on Zapier.

You can now automate web scraping workflows with other apps through Zapier. The integration uses the Scrape API v2 endpoint to scrape single webpages.

The Zapier app includes actions for:

• Scrape a single webpage

The integration supports all v2 features including running prompts on scraped content to extract specific information or format the output. You can also specify output formats (markdown, cleaned text, or HTML) and use CSS selectors to clean unwanted elements.

Read How to integrate no-code webcrawler in Zapier

Connect your WebcrawlerAPI account and start building automated workflows.

WebcrawlerAPI on Zapier

Scrape API v2 is now live.

Easier and straight forward. The new version lets you run a prompt on the page. You can get results in markdown, cleaned text, or HTML. Scraping is now in synchronous mode, with a single API call.

The new endpoint is at https://api.webcrawlerapi.com/v2/scrape. See the API Reference for details.

What's new?

• Scraping is now sync, with a single API call.
• You can remove parts of the page using CSS selectors.
• You can get results in markdown, cleaned text, or HTML.
• You can run a prompt on the page.
• Error handling is improved.

Postman collection is here to help you with WebcrawlerAPI testing.

Methods included in the collection:

• POST Start Crawl Job
• GET Get Job Status
• PUT Cancel Job
• GET Get Job URLs
• POST Start Scrape
• GET Get Scrape Status and Data
• GET Get Scrape Meta
• GET Get Scrape Result

Get your API key and try it out.

WebcrawlerAPI Postman Collection

WebcrawlerAPI now supports direct export to any S3 compatible storage.

What's new:

• Export crawl results directly to Amazon S3 buckets (or any S3 compatible storage: Cloudflare R2, DigitalOcean Spaces, Wasabi, Backblaze B2, etc.)
• Simple setup with API keys and bucket information
• We don't store your keys after job ends

How it works:

When starting a job via API, just add several parameters, like access_key_id, secret_access_key and a few others. Crawled data will be placed under the specified path. Your keys will be deleted after the job ends. Read Upload to S3 docs for detailed information.

Organizations has been added to WebcrawlerAPI. This feature lets multiple team members use the same API account with different access levels.

What's new:

• Organizations are now automatically created for all accounts
• All existing users have been assigned the "OWNER" role
• New "DEVELOPER" role with limited access:
  • Can use API and see usage statistics
  • Cannot access billing information
  • Cannot add or manage team members

How it works:

To add team members, go to your dashboard and click the "Invite member" button. You can assign roles based on what each person needs to do. This lets developers use the API without seeing billing details or changing the team.

We're thrilled to announce the release of our official LangChain integration! The new webcrawlerapi-langchain package makes it seamless to incorporate WebcrawlerAPI's powerful web crawling capabilities into your LangChain document processing pipelines.

Key Features:

• 🚀 Simple integration with LangChain's document loaders
• 📄 Multiple content formats (markdown, cleaned text, HTML)
• ⚡️ Async and lazy loading support
• 🔄 Built-in retry mechanisms, proxies and error handling
• 🎯 Configurable URL filtering with regex patterns

Quick Start:

pip install webcrawlerapi-langchain

from webcrawlerapi_langchain import WebCrawlerAPILoader

loader = WebCrawlerAPILoader(
    url="https://example.com",
    api_key="your-api-key",
    scrape_type="markdown"
)
documents = loader.load()

Perfect for:

• Building AI-powered knowledge bases
• Creating document QA systems
• Training custom language models
• Processing web content for LLM applications

Need an integration example?

Check our WebcrawlerAPI examples

Check out our LangChain SDK documentation for detailed usage instructions and examples. Start building powerful AI applications with web data today!

We're excited to announce that all new WebcrawlerAPI accounts now receive a $10 evaluation balance for a 7-day trial period! This initiative allows new users to thoroughly test our API capabilities without any upfront commitment.

What's included:

• $10 trial funds automatically added to new accounts
• Complete API access during 7-day evaluation period
• Start immediately with no credit card required
• Full access to all standard API features

The new trial balance makes it easier than ever to evaluate WebcrawlerAPI and test its capabilities for your projects.

• Pagination for jobs and job items
• Download button now has a progress and file size
• Graphs now more interactive

Major dashboard improvements 💫

• Enhanced login with email form:
  • Implemented rate limiting for magic link emails
  • Improved user experience and security
• Dashboard page enhancements:
  • Added time period toggles (24h, 7d, 15d, 30d)
  • Implemented total counter for each period
  • Enhanced graphs for funds spent and crawled pages
• New dedicated billing page:
  • Comprehensive payment history
  • Detailed payment usage tracking for all time

Major Update 🚀

• Integrated proxy management system:
  • All proxies are now handled internally
  • Included in the standard pricing
  • Significantly improved success rates
  • Enhanced protection against anti-bot measures
  • No additional setup required from users

Launched free llmstxt Generator Tool that helps create standardized llms.txt files for documenting AI models in your projects. You can learn more about the llms.txt standard in our detailed guide.

Major WebcrawlerAPI update: Comprehensive error handling system implementation

• Added two-level error handling system: job level and job item level errors
• New job level error codes:
  • insufficient_balance for balance-related issues
  • invalid_request for malformed requests
  • internal_error for system-level issues
• New job item level error codes:
  • host_returned_error for non-200 HTTP responses
  • website_access_denied for 403 responses
  • name_not_resolved for DNS resolution failures
  • internal_error for system-level issues
• Each error now includes detailed error messages and specific error codes for better debugging

Major improvements to our headless browser implementation for enhanced web scraping capabilities:

• Improved anti-bot protection bypass mechanisms
• Enhanced blocking of non-essential content:
  • Advertisement content filtering
  • Cookie consent banner removal
  • Other non-page-content elements blocking
• These updates result in cleaner data extraction and improved scraping reliability

The issue lasted for 9 hours but was not related to crawling. The root cause was a network issue affecting the monitoring server. Because the monitoring server was unavailable to the main job manager, each job report had to wait several minutes for a timeout response from the monitoring server.

As a result, the processing time for each job increased, and the job queue grew to several thousand jobs.

The incident has now been resolved. We are continuously working on improving our monitoring system to prevent similar issues in the future.

A status page link has been added to the website footer. The current status of WebCrawlerAPI services can now be checked at status.webcrawlerapi.com.

A changelog page has been added to the website. This page tracks all the changes, improvements, and fixes to WebCrawlerAPI.

A new tool Webpage to Markdown has been added. This tool converts any documentation or website into a beautiful Markdown file. It is free and does not require an API key. It can crawl up to 100 pages.

PDF content rendering has been implemented. Text content can now be extracted from PDF files. When a website contains a PDF file, its content will be extracted and returned in the response as page content.