Introducing the Crawling Agent (Wagent)
Meet the Crawling Agent — an AI-powered service that browses websites, follows links, and returns structured JSON, all from a natural-language prompt.
What it does
Unlike standard scraping that fetches a single URL, the agent acts like a researcher: it can perform web searches, read pages, decide which links are relevant, follow them, and extract the data you asked for. It autonomously decides how to find and extract the requested information. You describe what you want; the agent figures out how to get it.
Use cases
- Find customers, partners, or contact info across a company's website
- Extract product lists, pricing tiers, or feature sets from competitor sites
- Gather job listings, articles, or announcements from sites without an API
- Verify published information (pricing, specs, availability) against your expectations
- Any other structured data extraction task you can describe in plain language
Key parameters
- prompt — natural language instruction describing what to extract
- max_spend_usd — spending cap per run (required)
- urls — seed URLs to start from
- output_schema — JSON Schema to enforce a strict response shape
- model — choose from models like openai/gpt-5.4-mini, anthropic/claude-sonnet-4.6, google/gemini-3.1-flash-lite-preview, and more
Pricing
You pay per LLM token used and per page scraped during a run. A typical task can cost anywhere from a fraction of a cent up to the max_spend_usd limit you set, depending on complexity. All spending is visible in your dashboard.
Check out the Crawling Agent documentation for prompt-writing tips, examples, and response details, or jump straight to the API reference.