Mar 28, 2025
- Major Update 🚀
- Integrated proxy management system:
- All proxies are now handled internally
- Included in the standard pricing
- Significantly improved success rates
- Enhanced protection against anti-bot measures
- No additional setup required from users
- Integrated proxy management system:
Mar 24, 2025
- Launched free llmstxt Generator Tool that helps create standardized llms.txt files for documenting AI models in your projects. You can learn more about the llms.txt standard in our detailed guide.
Mar 21, 2025
- Major WebcrawlerAPI update: Comprehensive error handling system implementation
- Added two-level error handling system: job level and job item level errors
- New job level error codes:
insufficient_balance
for balance-related issuesinvalid_request
for malformed requestsinternal_error
for system-level issues
- New job item level error codes:
host_returned_error
for non-200 HTTP responseswebsite_access_denied
for 403 responsesname_not_resolved
for DNS resolution failuresinternal_error
for system-level issues
- Each error now includes detailed error messages and specific error codes for better debugging
Mar 14, 2025
- Major improvements to our headless browser implementation for enhanced web scraping capabilities:
- Improved anti-bot protection bypass mechanisms
- Enhanced blocking of non-essential content:
- Advertisement content filtering
- Cookie consent banner removal
- Other non-page-content elements blocking
- These updates result in cleaner data extraction and improved scraping reliability
Mar 06, 2025
The issue lasted for 9 hours but was not related to crawling. The root cause was a network issue affecting the monitoring server. Because the monitoring server was unavailable to the main job manager, each job report had to wait several minutes for a timeout response from the monitoring server.
As a result, the processing time for each job increased, and the job queue grew to several thousand jobs.
The incident has now been resolved. We are continuously working on improving our monitoring system to prevent similar issues in the future.
Mar 03, 2025
- A status page link has been added to the website footer. The current status of WebCrawlerAPI services can now be checked at status.webcrawlerapi.com.
Feb 22, 2025
- A changelog page has been added to the website. This page tracks all the changes, improvements, and fixes to WebCrawlerAPI.
Feb 19, 2025
- A new tool Webpage to Markdown has been added. This tool converts any documentation or website into a beautiful Markdown file. It is free and does not require an API key. It can crawl up to 100 pages.
Feb 18, 2025
- PDF content rendering has been implemented. Text content can now be extracted from PDF files. When a website contains a PDF file, its content will be extracted and returned in the response as page content