Google goto Links Now Resolve

Scraping links that come from Google search results is more reliable now. Google results increasingly contain google.com/goto?url=... redirect links instead of direct page URLs, and we now follow them to the real page before scraping.

What changed on Google's side

Google added a limit on how fast these goto links can be followed. When many links from the same results page are opened in a short time, Google stops redirecting right away. It shows a "verifying your request" page and asks the client to wait. The wait grows with every link from the same page, from a few seconds up to a minute or more. Changing IP addresses or the search query does not remove it. Until now, those links often failed.

What WebCrawlerAPI does now

• We follow the redirect to the final page and scrape that page. The url you sent stays unchanged in the response.
• If Google asks us to wait, we wait for up to 60 seconds in total, then continue.
• If the link cannot be resolved in that time, the item fails fast with the webpage_non_success error. We never scrape the raw goto link, and a failed item is not charged.

Tips

• If you collect many links from one results page, expect the later ones to take longer, since Google makes each wait longer than the last.
• Sending the direct page URLs, when your search source can provide them, skips the wait completely.

See the error codes for webpage_non_success.