WebcrawlerAPI now supports direct export to any S3 compatible storage.
What's new:
- Export crawl results directly to Amazon S3 buckets (or any S3 compatible storage: Cloudflare R2, DigitalOcean Spaces, Wasabi, Backblaze B2, etc.)
- Simple setup with API keys and bucket information
- We don't store your keys after job ends
How it works:
When starting a job via API, just add several parameters, like access_key_id, secret_access_key and a few others. Crawled data will be placed under the specified path. Your keys will be deleted after the job ends. Read Upload to S3 docs for detailed information.