AI Scraper

Combine Scraping and AI execution in one API

Run the OpenAI prompt over the webpage content and return a JSON! Just send a URL and a prompt to the API and get the results in seconds.

Forget about tokens, models, and training data. All you need is the URL and prompt to get the results.

Stop worrying about subscription and calculate your token usage. This API is pay-per-usage for a fixed price!

How can I use AI Scraper?

There are a lot of possible use cases. For example:

🕵️‍♂️ Competitor research:
- Analyze SEO keywords of competitors
- Follow pricing
- Landing page changes
- New shipped features
🔎 SEO optimisation - analyse your blog posts, articles and webpage content to optimise them for Search Engines
🕷️ Web Scraping - extract structured data in the format you want
✍️ Check the grammar on the webpage
👥 Analyze engagements via comments and posts
💫 Follow our blog posts for more use cases and prompts

Input

As simple as it gets, just send a URL and a prompt to the API and get the results in seconds.

url (required) - URL to scrape and convert to markdown
prompt (required) - Detailed instruction to the AI model what task to perform on the scraped webpage content

Example:

  {
    "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "prompt": "If this is markdown does not contain a book information (like description, price, etc) just return empty json .If it is then in json return all basic properties: image_link, title, price, rating, description, sub object: product information."
  }

How it works

We download the webpage content using the URL provided and then run the OpenAI prompt over the content.

Output

The response depends on your prompt. You can ask the AI to extract any information from the webpage content in JSON.

Example of extracting book information from the webpage content:

{
    "image_link": "../../media/cache/08/e9/08e94f3731d7d6b760dfbfbc02ca5c62.jpg",
    "title": "Tipping the Velvet",
    "price": "£53.74",
    "rating": null,
    "description": "Erotic and absorbing...Written with starling power.--The New York Times Book Review Nan King, an oyster girl, is captivated by the music hall phenomenon Kitty Butler, a male impersonator extraordinaire treading the boards in Canterbury. Through a friend at the box office, Nan manages to visit all her shows and finally meet her heroine. Soon after, she becomes Kitty's dresser and the two head for the bright lights of Leicester Square where they begin a glittering career as music-hall stars in an all-singing and dancing double act. At the same time, behind closed doors, they admit their attraction to each other and their affair begins.",
    "product_information": {
        "upc": "90fa61229261140a",
        "product_type": "Books",
        "price_excl_tax": "£53.74",
        "price_incl_tax": "£53.74",
        "tax": "£0.00",
        "availability": "In stock (20 available)",
        "number_of_reviews": 0
    }
}

Legal

This web scraping tool is designed to collect only publicly available data from websites, ensuring that no private information such as emails, phone numbers, or other sensitive details is accessed or gathered. It is important to note that the responsibility for how the collected data is used lies entirely with the user of the tool. Users must ensure that their actions comply with relevant laws, regulations. The owner of the Scraper does not control or influence the use of the data and is not liable for any misuse by the user.