Actions

Actions allow you to specify post-processing operations that should be performed on your crawled data. They enable you to automate the entire workflow from crawling to storage or further processing.

Available Actions

Currently, the following actions are supported:

• S3 Upload: Upload the crawled data directly to Amazon S3 or any S3-compatible storage service.

Usage

To use actions, include an actions array in your crawl request. Each action is an object with a type field that specifies the action type, along with action-specific parameters.

{
  "url": "https://example.com",
  "scrape_type": "cleaned",
  "actions": [
    {
      "type": "upload_s3",
      // action-specific parameters
    }
  ]
}

Multiple Actions

You can specify multiple actions to be performed sequentially. For example, you could upload the data to multiple storage locations.

{
  "url": "https://example.com",
  "scrape_type": "cleaned",
  "actions": [
    {
      "type": "upload_s3",
      // S3 parameters
    },
    {
      "type": "another_action",
      // parameters for another action
    }
  ]
}

Action Results

For details on specific actions and their parameters, refer to the individual action documentation pages.