The .NET SDK is published on NuGet as WebCrawlerApi.
Obtain an API Key
To use the WebCrawler API, you need to obtain an API key. You can do this by signing up for a free account.
Installation
bash
dotnet add package WebCrawlerApiRequires .NET 9.0 or higher.
Scrape a Page
ScrapeAndWaitAsync starts a scrape and waits until the result is ready (up to 400 seconds), so slow pages are not cut off by the 119 second limit of synchronous requests.
csharp
using WebCrawlerApi;
using WebCrawlerApi.Models;
using var client = new WebCrawlerApiClient("YOUR_API_KEY");
var result = await client.ScrapeAndWaitAsync(new ScrapeRequest
{
Url = "https://example.com",
OutputFormat = "markdown"
});
if (result.Success)
{
Console.WriteLine($"Page title: {result.PageTitle}");
Console.WriteLine(result.Markdown);
}
else
{
Console.WriteLine($"Error: {result.ErrorCode} - {result.ErrorMessage}");
}Crawl a Website
CrawlAndWaitAsync crawls a website and returns all pages once the job is finished.
csharp
using WebCrawlerApi;
using var client = new WebCrawlerApiClient("YOUR_API_KEY");
var job = await client.CrawlAndWaitAsync(
url: "https://example.com",
scrapeType: "markdown",
itemsLimit: 10
);
Console.WriteLine($"Job completed with status: {job.Status}");
foreach (var item in job.JobItems)
{
Console.WriteLine($"{item.Title} - {item.OriginalUrl}");
var content = await item.GetContentAsync();
Console.WriteLine(content);
}For all methods and options, see the package on NuGet or the GitHub repository.