Ruby

Obtain an API Key

To use the WebCrawler API, you need to obtain an API key. You can do this by signing up for a free account.

Installation

bash

gem install webcrawlerapi

Or add it to your Gemfile:

ruby

gem "webcrawlerapi"

The source code is on GitHub and the gem is on RubyGems.

Requirements

Ruby 3.0 or higher
No third-party dependencies

Usage

Request options are passed as a Hash. Symbol and string keys both work, and the keys are the same as in the API reference (snake_case).

Scrape a page

scrape starts the scrape and waits for the result.

ruby

require "webcrawlerapi"

client = WebCrawlerAPI::Client.new("YOUR_API_KEY")

res = client.scrape(url: "https://example.com", output_formats: ["markdown"])

if res.error_code
  puts "#{res.error_code}: #{res.error_message}"
else
  puts res.markdown
end

A scrape that fails on the target side (for example blocked_by_robots_txt) is returned as a ScrapeResponse with error_code set, not raised as an exception.

Synchronous Crawling

crawl waits for the crawl to complete and returns the whole job.

ruby

job = client.crawl(
  url: "https://example.com",
  output_formats: ["markdown"],
  items_limit: 10
)

puts "Job completed with status: #{job.status}"

job.job_items.each do |item|
  puts "Page title: #{item.title}"
  puts "Original URL: #{item.original_url}"

  # Content in the requested output format.
  # Returns nil if the job or the item is not in "done" status.
  content = item.get_content
  puts content[0, 200] if content
end

Asynchronous Crawling

crawl_async returns a job ID immediately, and you check the status later. You can also use webhooks.

ruby

started = client.crawl_async(url: "https://example.com", items_limit: 10)
puts "Crawling job started with ID: #{started.id}"

job = client.get_job(started.id)
while [WebCrawlerAPI::JobStatus::NEW, WebCrawlerAPI::JobStatus::IN_PROGRESS].include?(job.status)
  sleep(job.recommended_pull_delay_ms.to_i / 1000.0)
  job = client.get_job(started.id)
end
puts "Job status: #{job.status}"

Batch scrape

Scrape a list of URLs in one async request. Pass an Array as url and scrape creates the batch and waits until it finishes.

ruby

batch = client.scrape(
  url: ["https://example.com", "https://example.com/pricing"],
  max_concurrency: 5
)

puts "#{batch.status}: #{batch.completed}/#{batch.total} done, #{batch.failed} failed"
batch.data.each { |item| puts "#{item['url']} #{item['status']} #{item['markdown_url']}" }

Method	Description
`scrape` (with an Array `url`) / `scrape_batch`	Create a batch and wait for it to finish (first 100 items)
`scrape_batch_async`	Create a batch and return its id right away
`get_batch`	Batch progress and one page of items (`limit:`, `cursor:` from `next`)
`get_batch_errors`	Failed URLs and URLs blocked by robots.txt
`cancel_batch`	Cancel a running batch; unstarted pages are not charged

See the batch scrape guide and the API reference.

Other methods

Method	Description
`scrape_async` / `get_scrape`	Start a scrape and fetch its state later
`get_job_markdown` / `get_job_markdown_content`	Combined markdown of a whole job (URL or text)
`markdown`	Clean article markdown of a single page
`run_agent` / `run_agent_async` / `get_agent_job`	Run an AI agent task

Request Parameters

scrape:

url (required): The target URL to scrape (or an Array of URLs for a batch)
output_formats: markdown, cleaned, html, links
prompt: Prompt to run on the scraped content (extra cost)
response_schema: JSON schema for structured output with prompt
clean_selectors: CSS selectors to remove from the output
main_content_only: Extract only the main content
max_age: Maximum age of cached content in seconds
webhook_url: URL to receive a notification when the scrape completes

crawl:

url (required): The seed URL
output_formats: markdown, cleaned, html, links (default: markdown)
items_limit: Maximum number of pages to crawl
whitelist_regexp / blacklist_regexp: Regular expressions for allowed or blocked URLs
max_depth: Maximum crawl depth
respect_robots_txt, main_content_only, keep_query_params: booleans
max_age: Maximum age of cached content in seconds
webhook_url: URL to receive a notification when the job completes

Client options

ruby

client = WebCrawlerAPI::Client.new("YOUR_API_KEY",
  open_timeout: 10,
  read_timeout: 60
)

# Limit the number of status polls in scrape, crawl and run_agent
job = client.crawl({ url: "https://example.com" }, 50)

Error Handling

API errors are raised as WebCrawlerAPI::Error. See Errors for all error codes.

ruby

begin
  client.crawl(url: "https://example.com")
rescue WebCrawlerAPI::Error => e
  puts "#{e.error_code} #{e.error_message} #{e.status_code}"
end

crawl, scrape with an Array url, and run_agent raise an error with code timeout when the job is still running after the maximum number of polls. Network failures are raised with code network_error.

Response Objects

Responses are read-only objects. Read a field as a method (job.status), by key (job[:status]) or convert to a Hash with to_h.

Job

ruby

job.id                        # Unique job identifier
job.status                    # Job status (new, in_progress, done, error)
job.url                       # Original crawl URL
job.created_at                # Job creation timestamp
job.finished_at               # Job completion timestamp
job.job_items                 # Crawled items
job.recommended_pull_delay_ms # Recommended delay between status checks

JobItem

ruby

item.id                    # Unique item identifier
item.original_url          # URL of the crawled page
item.title                 # Page title
item.status                # Item status
item.page_status_code      # HTTP status code
item.markdown_content_url  # URL to markdown content
item.cleaned_content_url   # URL to cleaned content
item.raw_content_url       # URL to raw HTML content
item.get_content           # Content in the job output format (nil if not "done")
item.get_markdown          # Markdown content
item.get_cleaned           # Cleaned content
item.get_html              # Raw HTML content