The idea for WebCrawlerAPI—an API where you can send a link and get the markdown for every page—came in 2023, right after GPT-3.5 was released. My friend and I wanted to build a support chatbot for SaaS, that could answer questions based on a real website content. There was nothing like that at that moment and the idea was new. But there was nothing to actually get the full content of the website - scrapers could only do it with one page. The use case was simply too fresh.
I hacked it with some scripts. In a few days we had a working prototype. The chatbot demo went nowhere. But the crawler stuck with me.
I released the first version of WebCrawlerAPI in 2024 and have worked on it every day since.
Build the best API possible - this is my life mission. Fast, accurate, reliable API with perfect markdown - is the goal. And my 15 years engineering background with complicated systems, APIs and big data helps me here. The main focus is on quality. When something breaks, I hear about it almost immediately.
I always answer myself in support chat. No AI chatbots (ironically). If you reach out there - make sure it is me, who wrote code, answering your question. This gives me a pulse - talk with real customers.
WebCrawlerAPI is an indie-built product. This gives the freedom to iterate fast, try ideas, and improve things quickly. But it also means every decision matters, because time and resources are limited.
No matter what, it is incredibly enjoyable to be on this journey and see customers using it every day.
Thanks for choosing WebCrawlerAPI. I will continue making it better every day.
