Go

Obtain an API Key

To use the WebCrawler API, you need to obtain an API key. You can do this by signing up for a free account.

Installation

bash

go get github.com/WebCrawlerAPI/webcrawlerapi-go-sdk

The source code is on GitHub.

Requirements

Go 1.21 or higher
No third-party dependencies

Usage

Every method takes a context.Context as the first argument, so you can cancel requests or set deadlines.

Scrape a page

Scrape starts the scrape and waits for the result.

package main

import (
	"context"
	"fmt"
	"log"

	webcrawlerapi "github.com/WebCrawlerAPI/webcrawlerapi-go-sdk"
)

func main() {
	client := webcrawlerapi.New("YOUR_API_KEY")

	res, err := client.Scrape(context.Background(), webcrawlerapi.ScrapeRequest{
		URL:           "https://example.com",
		OutputFormats: []string{"markdown"},
	})
	if err != nil {
		log.Fatal(err)
	}
	if res.IsError() {
		log.Fatalf("%s: %s", res.ErrorCode, res.ErrorMessage)
	}
	fmt.Println(res.Markdown)
}

A scrape that fails on the target side (for example blocked_by_robots_txt) is returned as a ScrapeResponse with ErrorCode set, not as a Go error. Check it with res.IsError().

Synchronous Crawling

Crawl waits for the crawl to complete and returns the whole job.

ctx := context.Background()

job, err := client.Crawl(ctx, webcrawlerapi.CrawlRequest{
	URL:           "https://example.com",
	OutputFormats: []string{"markdown"},
	ItemsLimit:    10,
})
if err != nil {
	log.Fatal(err)
}

fmt.Println("Job completed with status:", job.Status)

for _, item := range job.JobItems {
	fmt.Println("Page title:", item.Title)
	fmt.Println("Original URL:", item.OriginalURL)

	// Content in the requested output format.
	// Returns "" if the item is not in "done" status.
	content, err := item.GetContent(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(content[:min(len(content), 200)])
}

Asynchronous Crawling

CrawlAsync returns a job ID immediately, and you check the status later. You can also use webhooks.

ctx := context.Background()

started, err := client.CrawlAsync(ctx, webcrawlerapi.CrawlRequest{
	URL:        "https://example.com",
	ItemsLimit: 10,
})
if err != nil {
	log.Fatal(err)
}
fmt.Println("Crawling job started with ID:", started.ID)

job, err := client.GetJob(ctx, started.ID)
for err == nil && (job.Status == webcrawlerapi.JobStatusNew || job.Status == webcrawlerapi.JobStatusInProgress) {
	time.Sleep(time.Duration(job.RecommendedPullDelayMs) * time.Millisecond)
	job, err = client.GetJob(ctx, started.ID)
}
if err != nil {
	log.Fatal(err)
}
fmt.Println("Job status:", job.Status)

Other methods

Method	Description
`ScrapeAsync` / `GetScrape`	Start a scrape and fetch its state later
`GetJobMarkdown` / `GetJobMarkdownContent`	Combined markdown of a whole job (URL or text)
`Markdown`	Clean article markdown of a single page
`RunAgent` / `RunAgentAsync` / `GetAgentJob`	Run an AI agent task

Request Parameters

ScrapeRequest:

URL (required): The target URL to scrape
OutputFormats: markdown, cleaned, html, links
Prompt: Prompt to run on the scraped content (extra cost)
ResponseSchema: JSON schema for structured output with Prompt
CleanSelectors: CSS selectors to remove from the output
MainContentOnly: Extract only the main content (*bool)
MaxAge: Maximum age of cached content in seconds (*int)
WebhookURL: URL to receive a notification when the scrape completes

CrawlRequest:

URL (required): The seed URL
OutputFormats: markdown, cleaned, html, links (default: markdown)
ItemsLimit: Maximum number of pages to crawl
WhitelistRegexp / BlacklistRegexp: Regular expressions for allowed or blocked URLs
MaxDepth: Maximum crawl depth (*int)
RespectRobotsTxt, MainContentOnly, KeepQueryParams: booleans (*bool)
MaxAge: Maximum age of cached content in seconds (*int)
WebhookURL: URL to receive a notification when the job completes

Optional numeric and boolean fields are pointers, because 0 and false are meaningful values. Use the webcrawlerapi.Ptr helper:

webcrawlerapi.CrawlRequest{
	URL:             "https://example.com",
	MaxDepth:        webcrawlerapi.Ptr(1),
	MainContentOnly: webcrawlerapi.Ptr(true),
}

Client options

client := webcrawlerapi.New("YOUR_API_KEY",
	webcrawlerapi.WithHTTPClient(&http.Client{Timeout: 30 * time.Second}),
)

// Limit the number of status polls in Scrape, Crawl and RunAgent
job, err := client.Crawl(ctx, req, webcrawlerapi.WithMaxPolls(50))

Error Handling

API errors are returned as *webcrawlerapi.Error. See Errors for all error codes.

_, err := client.Crawl(ctx, req)

var apiErr *webcrawlerapi.Error
if errors.As(err, &apiErr) {
	fmt.Println(apiErr.ErrorCode, apiErr.ErrorMessage, apiErr.StatusCode)
}

Crawl and RunAgent return an error with code timeout when the job is still running after the maximum number of polls.

Response Objects

Job

job.ID                      // Unique job identifier
job.Status                  // Job status (new, in_progress, done, error)
job.URL                     // Original crawl URL
job.CreatedAt               // Job creation timestamp
job.FinishedAt              // Job completion timestamp
job.JobItems                // Crawled items
job.RecommendedPullDelayMs  // Recommended delay between status checks

JobItem

item.ID                  // Unique item identifier
item.OriginalURL         // URL of the crawled page
item.Title               // Page title
item.Status              // Item status
item.PageStatusCode      // HTTP status code
item.MarkdownContentURL  // URL to markdown content
item.CleanedContentURL   // URL to cleaned content
item.RawContentURL       // URL to raw HTML content
item.GetContent(ctx)     // Content in the job output format ("" if not "done")
item.GetMarkdown(ctx)    // Markdown content
item.GetCleaned(ctx)     // Cleaned content
item.GetHTML(ctx)        // Raw HTML content