WebcrawlerAPI Rate Limits

To ensure ethical web scraping practices and prevent excessive load on target websites, we implement rate limiting mechanisms. These limits help maintain a balance between efficient data collection and responsible website interaction.

Rate Limit Rules

Per-Account Limits

• Each account is limited to 5 parallel threads for concurrent scraping operations
• This limit applies regardless of the target website

Per-Website Limits

• Each website has a shared limit of 5 parallel threads across all accounts
• This limit is shared among different accounts scraping the same website

Example Scenario

Let's consider a situation with two different accounts scraping the same website:

1. Account A starts scraping example.com using 3 parallel threads
2. Account B attempts to scrape example.com at the same time
3. Since example.com already has 3 threads in use:
   • Account B can only use up to 2 additional threads (5 total - 3 in use = 2 available)
   • This is true even if Account B's personal thread limit hasn't been reached

This shared website limit ensures that no single website receives excessive load, even when multiple accounts are scraping simultaneously.