Sitemap Crawling, Job Cost Visibility, and Dashboard Filters

A set of quality-of-life improvements landed this month across the API and dashboard.

Sitemap-assisted crawl discovery

Crawl jobs now automatically fetch and parse sitemap.xml when starting a new crawl. Pages found in the sitemap are added to the initial link queue, so the crawler reaches more pages earlier - without any changes to your requests.

Job cost in the dashboard

Each job in the jobs list now shows how much it cost. Useful for tracking spend per crawl or scrape run without having to cross-reference billing statements.

Job status filter

You can now filter your jobs list by status — done, processing, failed, and others. Quickly find the runs you care about without scrolling through everything.

Sync scrape timeout extended

The synchronous /v2/scrape endpoint now waits up to 3 minutes (previously 2 minutes) before timing out. Heavy pages and slower sites are less likely to return a timeout error.

Tax-inclusive billing

Top-up payments now go through Stripe Checkout, which automatically calculates and applies taxes based on your billing region.