September 25, 2025

Sep '25 Updates

Google News feed scraping: Fixed and now working reliably
New Integration Page: Added to dashboard showing all available WebCrawlerAPI integrations including code, no-code, and storage options
Webhook status tracking: Now visible in dashboard with "Resend" button to retry failed webhook deliveries
Infrastructure optimizations: Enhanced crawling and scraping performance
WWW handling: Websites with and without www subdomain are now processed correctly
Model upgrade: Switched to google/gemini-2.5-flash-lite for prompt processing - significantly faster than OpenAI models
Main content extraction: New parameter to extract only useful text from webpages, perfect for blog posts and articles
Milestone achieved: WebCrawlerAPI crossed 750K total crawled pages this month