The best vector databases for a support bot or AI search over your website and docs are not the ones that win billion-vector benchmarks. A docs knowledge base is a small-data problem: a few thousand pages, tens of thousands of chunks. That changes which database makes sense.
I'm Andrew. Here I compare 14 vector databases, with prices for the free, cheap, local and hosted ones, plus a small benchmark I ran on real docs chunks. Prices and free-tier limits were re-checked on vendor pricing pages on November 11, 2026.
Free and Cheapest Vector Databases at a Glance
- Free to self-host (open source): pgvector, Qdrant, Chroma, Milvus, LanceDB, sqlite-vec, DuckDB vss and OpenSearch. You pay only for the machine.
- Free cloud tiers: Pinecone Starter (2 GB), Qdrant Cloud (1 GB RAM, 4 GB disk), Weaviate Cloud (100,000 objects), Zilliz Cloud (5 GB), Supabase (500 MB Postgres with pgvector), Neon (1 GB per project), MongoDB Atlas M0 (512 MB), Redis (30 MB), Chroma Cloud ($5 credits) and Cloudflare Vectorize (5M stored dimensions).
- Good for a side project: Pinecone Starter or Zilliz Cloud free if you want no server, a free Supabase project if you like SQL, or a local file with sqlite-vec or LanceDB.
Cheapest paid managed plans, from lowest entry price (November 11, 2026):
| Service | Entry price | How it bills |
|---|---|---|
| Zilliz Cloud Serverless | From $0 | Usage |
| Chroma Cloud | $0 + usage | Usage, $5 free credits |
| Neon (Postgres + pgvector) | $0.106/CU-hour, $0.35/GB-month | Usage |
| Redis Cloud Essentials | About $5/month ($0.007/hour) | Hourly |
| Cloudflare Vectorize | $5/month (Workers Paid) | Plan + usage |
| MongoDB Atlas Flex | $8 to $30/month ($0.011/hour) | Hourly, capped |
| turbopuffer Launch | $16/month minimum | Usage with a floor |
| Pinecone Builder | $20/month | Flat |
| Supabase Pro | $25/month | Plan |
| Weaviate Cloud Flex | From $45/month | Minimum + usage |
| Elastic Cloud Hosted | As low as $99/month | Plan |
| Zilliz Cloud Dedicated | From $126/month | Dedicated cluster |
Qdrant Cloud is usage-based with no list price on its page. Free-tier limits, gotchas and whether 100,000 chunks fit are in the pricing section below. Self-hosted vs hosted trade-offs are in their own section.
Pick by Situation
- Already on Postgres: pgvector.
- Zero ops, managed only: Pinecone.
- Free self-hosting with a managed option: Qdrant.
- Hybrid search is the main need: Weaviate.
- Local, no server: sqlite-vec, LanceDB or Chroma.
- Cheapest managed to start: no single winner. Zilliz Serverless (from $0) and Chroma Cloud ($0 + usage) have no fixed floor, Redis Essentials and Cloudflare Vectorize (Workers Paid) are about $5/month, turbopuffer is $16/month. Check the limits in the pricing table.
- Huge scale: Milvus or Zilliz Cloud (Milvus Lite also runs as a local folder for prototypes).
- Already running Elasticsearch or OpenSearch: stay there.
At the sizes tested here (2,159 chunks, plus the 100k-row run in the pgvector tutorial), every store I ran answered in milliseconds. Pick on ops burden, filters and price.
A disclosure: I build WebCrawlerAPI, so ingestion is the part I know best. I ran pgvector on a real crawl earlier, and I ran the local benchmark below on 2,159 chunks myself. Everything about managed services is read from official pages: features on October 24, 2026, prices re-checked on November 11, 2026. GitHub stars and several open source licences come from the GitHub API on the same day. If you only need markdown from a site, the free website to markdown tool does that. The full pipeline is in website to markdown for knowledge bases.
Vector Database List (A to Z)
- Chroma: open source embedding database, in-process, as a server in Docker or as a cloud service. Tutorial: Chroma in Docker with metadata filters.
- Cloudflare Vectorize: managed, for Workers apps. Tutorial coming.
- DuckDB vss: HNSW index extension for DuckDB. Tutorial coming.
- Elasticsearch / OpenSearch: search engines with vector fields. Tutorial: hybrid BM25 and kNN search on Elasticsearch and OpenSearch.
- LanceDB: embedded library, data in files. Tutorial: LanceDB with local files and an S3-compatible store.
- Milvus / Zilliz Cloud: open source for very large scale, plus a managed cloud. Milvus Lite also runs as a local folder for prototypes. Tutorial: Milvus Lite and Zilliz Cloud.
- MongoDB Atlas: $vectorSearch inside Atlas. See the MongoDB Atlas Vector Search tutorial.
- pgvector: Postgres extension. Tutorial below, in the pgvector section.
- Pinecone: managed, closed source. Tutorial: Pinecone with integrated embedding.
- Qdrant: open source in Rust, plus a managed cloud. Tutorial: Qdrant in Docker with payload filters.
- Redis: vector search in the Redis Query Engine. Tutorial coming.
- sqlite-vec: SQLite extension. Tutorial.
- turbopuffer: managed, object-storage based. Tutorial coming.
- Weaviate: open source with hybrid search, plus a managed cloud. Tutorial with measured hybrid search.
How Big Is a Knowledge Base, Really?
The rough maths is pages x chunks per page x bytes per vector, using 1,536 dimensions (the size of OpenAI's text-embedding-3-small). A 1,536-dim float32 vector is 6,152 bytes in pgvector (4 x dimensions + 8), so about 6 KB.
Chunks per page varies a lot. I measured about 4.5 on one small crawl, 6 on 50 htmx pages (298 chunks), and about 15 on the FastAPI docs (2,159 chunks from 147 pages). The table uses 10 as the middle:
| Site size | Chunks (est.) | Raw vectors (est.) |
|---|---|---|
| 500 pages | 5,000 | ~31 MB |
| 5,000 pages | 50,000 | ~308 MB |
| 50,000 pages | 500,000 | ~3.1 GB |
Indexes and text come on top. In the new benchmark, the raw vectors were 13.3 MB and the stores ranged from 11.8 to 38.7 MB, so 0.9x to 2.9x the raw vectors in total, text and metadata included. Most docs sites and help centers sit in the first two rows. "Scales to 10 billion vectors" does not help you decide.
What Actually Matters
- Metadata filters. By URL, docs section, version and language. A weak filter returns fewer results than asked for.
- Upsert and delete by source URL. When a page changes, its old chunks go and new ones come in. Cadence is covered in how often to re-crawl a website for an AI knowledge base.
- Hybrid search. Pure vector search is weak on exact strings like ERR_429 or function names. Keyword search handles those, which is why hybrid search matters.
- Ops and price at small scale. A $50 monthly minimum is a lot for 300 MB of vectors. A self-hosted cluster is a lot of work for two people.
Chunking is in chunking strategies for knowledge base content. Which embedding model to use is measured in Best Embedding Model for Website Docs.
When You Don't Need a Vector Database
A few hundred to a few thousand pages, mostly exact terms: keyword search (Postgres full-text, SQLite FTS5, Elasticsearch) may be enough. Under a few thousand chunks, in-memory search or FAISS works too. Add vectors when users ask questions in words the docs do not use.
Vector Database Comparison Table
Read from vendor pages on October 24, 2026. A dash means not readable on an official page that day, not "no".
| Database | Type | Hosting | Free self-host | Free cloud tier | Hybrid | Metadata filters | Max dims | Best for |
|---|---|---|---|---|---|---|---|---|
| pgvector | Postgres extension | Self-host; hosted via Supabase | Yes (PostgreSQL License) | Via Supabase (hosted Postgres) | DIY with full-text | SQL WHERE | 2,000 indexed, 16,000 stored | Teams on Postgres |
| Pinecone | Managed | AWS, Azure, GCP | No | Starter | Yes | Metadata filters | 20,000 | Zero ops |
| Qdrant | Open source + cloud | Self-host or Qdrant Cloud | Yes (Apache-2.0) | 1 GB RAM, 4 GB disk | Yes (dense + sparse, BM25) | Payload filters | 65,536 | Self-hosting, filters |
| Weaviate | Open source + cloud | Self-host or Weaviate Cloud | Yes (BSD-3, mixed) | 100,000 objects | Yes (BM25F) | Pre-filtering | - | Hybrid-heavy search |
| Chroma | Open source + cloud | Self-host or Chroma Cloud | Yes (Apache-2.0) | $0 + usage, $5 credits | Yes | where filters | - | Prototypes |
| Milvus / Zilliz | Open source + cloud | Self-host or Zilliz Cloud | Yes (Apache-2.0) | 5 GB | Yes (BM25 + dense) | - (not read) | 32,768 | Very large corpora (Lite for prototypes) |
| Elasticsearch / OpenSearch | Search engine | Self-managed or Elastic Cloud | Yes (mixed / Apache-2.0) | Trial only | Native | OpenSearch yes; Elasticsearch - | 4,096 / 16,000 | Existing search stack |
| LanceDB | Embedded library | In your app; LanceDB Cloud | Yes (Apache-2.0) | - | Yes | SQL where | - | Local apps |
| turbopuffer | Managed | Managed | No | None | Yes (vector + BM25) | Yes | 10,752 | Usage-based cost |
| sqlite-vec | SQLite extension | In process | Yes (MIT / Apache-2.0) | n/a | No built-in | Metadata columns in WHERE | - | Tiny local apps |
| DuckDB vss | DuckDB extension | In process | Yes (MIT) | n/a | None found | SQL WHERE (not read in docs) | - | Analytics + vectors |
| MongoDB Atlas | Managed | Atlas | - | M0, 512 MB | Yes | $vectorSearch pre-filter | - | Teams on Mongo |
| Cloudflare Vectorize | Managed | Cloudflare | No | Workers Free | Not documented | Needs a metadata index | 1,536 | Workers apps |
| Redis | In-memory + cloud | Self-host or Redis Cloud | Yes (RSALv2 / SSPLv1 / AGPLv3) | 30 MB | Filters with KNN | Text, numeric, tag filters | - | Teams on Redis |

Filters are read from each vendor's docs, except Milvus, DuckDB and Elasticsearch, where I did not read the filter pages. Limits worth knowing: Vectorize needs a metadata index per filtered field (10 per index), pgvector applies filters after an approximate index scan unless iterative scans are on (0.8.0+), and sqlite-vec allows metadata columns in WHERE but not auxiliary ones. Licences: Weaviate is BSD-3 outside its wl directory, Elasticsearch is triple licensed, and Redis 8 is RSALv2, SSPLv1 or AGPLv3 (Redis itself says RSALv2 is not an open source licence). Atlas self-managed products were not checked.
Free and Cheap Vector Databases
Free Forever (Self-Hosted)
pgvector, Qdrant, Chroma, Milvus, LanceDB, sqlite-vec, DuckDB vss and OpenSearch cost nothing to license. You pay for the machine and the time. For a small knowledge base, the cheapest setup is usually an existing Postgres or a file on disk. On a server, the cheapest self-hosted option is Qdrant or Chroma in one Docker container, or pgvector in the Postgres you already run.
Free Cloud Tiers and Cheapest Paid Plans
Read from vendor pricing pages on November 11, 2026.
| Store | Free cloud tier | Cheapest paid | 100k chunks (~615 MB raw) in the free tier? |
|---|---|---|---|
| Supabase (pgvector) | 500 MB database, paused after 1 week idle, 2 active projects | Pro from $25/month | No, raw vectors alone exceed 500 MB |
| Neon (per Neon pricing page) | 1 GB storage per project, 100 CU-hours/month | Usage-based: $0.106/CU-hour, $0.35/GB-month | Tight: 615 MB raw plus index |
| Pinecone | Starter: 2 GB, 2M write units, 1M read units/month | Builder $20/month, Standard $50/month minimum | Storage yes, units are the limit |
| Qdrant Cloud | 0.5 vCPU, 1 GB RAM, 4 GB disk | - (usage-based, no price on page) | Tight with the index in RAM |
| Weaviate Cloud | 100,000 objects, 1 GB memory, 10 GB disk | Flex from $45/month | Exactly at the object limit |
| Chroma Cloud | $0 + usage, $5 credits | Team $250/month + usage | Yes, usage billed |
| Zilliz Cloud | 5 GB, 2.5M vCUs/month, 5 collections | Serverless from $0; Dedicated Standard from $126/month, Enterprise from $197/month | Yes |
| Elastic Cloud | Free trial only | Hosted Standard "as low as" $99/month | No permanent free tier |
| turbopuffer | None | Launch $16/month minimum, Scale $256/month | n/a |
| MongoDB Atlas | M0, 512 MB | Flex $0.011/hour, billed $8 to $30/month | No |
| Cloudflare Vectorize | 5M stored dimensions | Workers Paid $5/month minimum | No, about 3,255 vectors |
| Redis | 30 MB | Essentials from $0.007/hour (about $5/month), Pro from $200/month | No |
LanceDB Cloud and Qdrant paid prices were not readable, so they are dashes. Neon figures are from its pricing page only. Chroma Team includes $100 of monthly usage that does not roll over.
Cost assumption box. 1,536 dims x 4 bytes + 8 = 6,152 bytes per vector. 10,000 chunks is about 61 MB raw, 100,000 about 615 MB. Total store size in my run was 0.9x to 2.9x the raw vectors, indexes and text included. Vectorize bills stored dimensions: 100,000 vectors is 153.6M dimensions, and the free 5M holds about 3,255 vectors. On Workers Paid (10M stored included, then $0.05 per 100M), that is roughly $0.07 of storage on top of the $5 plan, before query charges.
Free-Tier Gotchas
- Pausing: Supabase free projects pause after a week idle.
- Unit caps: Pinecone Starter limits writes and reads, not only storage. Chroma pauses at the usage limit.
- Minimums: turbopuffer, Standard Pinecone and Redis Pro have monthly floors.
- RAM-bound: Qdrant's free node has 1 GB RAM.
- Dimension caps: Vectorize stops at 1,536, so larger embeddings do not fit.
Self-Hosted vs Managed Vector Databases
Self-hosting costs hours: backups, upgrades, monitoring and high availability. Managed costs money and some lock-in, but removes those hours. The data does not move if you keep source markdown, so lock-in is mostly the query code.
- "Data must stay in my network": pgvector, Qdrant, Weaviate, Milvus, OpenSearch.
- "No ops": Pinecone, Zilliz, Qdrant Cloud, Weaviate Cloud, turbopuffer, Chroma Cloud.
- "Already have Postgres": pgvector, on your server or on Supabase.
Which hosting fits a docs knowledge base:
| Self-hosted | Hosted (managed) | |
|---|---|---|
| Licence cost | $0 for the open source engines | Free tier, then from about $5 to $126/month to start |
| Your work | Backups, upgrades, monitoring, HA | API keys and a billing alert |
| Data location | Your network | Vendor cloud and region |
| Good picks | pgvector, Qdrant, Weaviate, Milvus, Chroma, OpenSearch | Pinecone, Zilliz Cloud, Qdrant Cloud, Weaviate Cloud, turbopuffer, Supabase |
Managed knowledge base services (Azure AI Search, Vertex AI, OpenAI and Gemini file search, Bedrock Knowledge Bases) skip the database choice altogether. Tutorials on them are coming.
Local and Embedded Vector Databases
No server, one file or folder: sqlite-vec, LanceDB, Chroma (in-process) and DuckDB vss. For one person, a CLI tool or a desktop app, there is no server to run. On 2,159 chunks, their stores took 11.8 to 25.4 MB, with ingest (insert plus index build) of 4.4 seconds or less. Charts are in the benchmark section below.
The catches: sqlite-vec is pre-v1 and brute force only. DuckDB vss keeps an HNSW index only in memory unless you enable experimental persistence, which DuckDB warns can lose or corrupt data after an unexpected shutdown. Sharing one index across several app instances is harder with any of them. The sqlite-vec tutorial builds a docs knowledge base in one SQLite file and adds a short DuckDB note.
The Best Vector Databases, One by One
pgvector

Pick this if your app already runs on Postgres. Filters are plain SQL and DELETE FROM chunks WHERE url = $1 is your freshness story. Where it hurts: tuning and index builds at tens of millions of rows. Setup and index details are in the pgvector tutorial.
Pinecone

Pick this if you want zero ops and accept a cloud-only, closed-source service.
- Dense, sparse and full-text indexes; metadata filter expressions.
- Free Starter, Builder $20/month, Standard from $50/month.
- Where it hurts: usage billing is harder to predict than a fixed server. Tutorial: Pinecone with integrated embedding.
Qdrant

Pick this if you want a dedicated database you can self-host for free and later move to a managed cloud.
- Rust, Apache-2.0. Payload filters, dense + sparse + BM25 hybrid.
- Free cloud node: 1 GB RAM, 4 GB disk, no high availability.
- Where it hurts: another service to run. Tutorial: Qdrant in Docker with payload filters.
Weaviate

Pick this if hybrid search is the main requirement.
- Vector plus BM25F keyword search, pre-filtering, free cloud tier of 100,000 objects.
- Flex from $45/month.
- Where it hurts: more concepts to learn. I have only run it locally, not in production. Tutorial: hybrid search on a crawled docs site.
Chroma

Pick this if you are prototyping and want an in-process store.
- Apache-2.0, runs in-process, as a server in Docker or as Chroma Cloud ($0 + usage, Team $250/month).
- Where it hurts: high-availability self-hosting is harder. Tutorial: Chroma in Docker with metadata filters.
Milvus / Zilliz Cloud

Pick this if you really have hundreds of millions of vectors.
- Apache-2.0. Zilliz Cloud is the managed version, with a free tier of 5 GB.
- Where it hurts: built for a scale far beyond 300 MB of vectors, although Milvus Lite runs as a local folder for prototypes. Tutorial: Milvus Lite and Zilliz Cloud.
Elasticsearch / OpenSearch

Pick this if you already run one for site search. Vectors sit next to your text index, and hybrid search is native. Where it hurts: a full search engine to operate if you do not already run one. Elastic Cloud Hosted Standard is listed as low as $99/month. See the Elasticsearch and OpenSearch tutorial. For a lighter engine, see the Meilisearch and Typesense tutorial.
LanceDB

Pick this if you want vector search inside your app with no server. Apache-2.0, files local or on object storage, SQL where filters, hybrid with rerankers. Where it hurts: sharing one index across servers. Tutorial: LanceDB with local files and an S3-compatible store.
turbopuffer
Pick this if you want a managed store with usage-based pricing.
- Object-storage based, vector plus BM25, up to 10,752 dense dimensions.
- No free tier. Launch has a $16/month minimum, Scale $256/month.
- Where it hurts: closed source, and every write is billed at least 10 KB. Tutorial coming.
Also Worth Knowing
- sqlite-vec: brute-force KNN in SQLite, metadata columns in WHERE. Pre-v1, so expect breaking changes.
- DuckDB vss: HNSW on fixed-size arrays. Persistence is experimental.
- MongoDB Atlas: $vectorSearch with pre-filters, M0 free at 512 MB. Good if you already use Mongo, see the MongoDB Atlas Vector Search tutorial.
- Cloudflare Vectorize: cheap and simple on Workers, capped at 1,536 dimensions and 20M vectors per index.
- Redis: FLAT, HNSW and SVS-VAMANA indexes, free cloud tier of 30 MB.
- Supabase: hosted Postgres, so the pgvector answer without running a server (a step-by-step run is in the Supabase pgvector tutorial). Neon is hosted Postgres too, but I read only its pricing page, so check its docs for pgvector support.
Most Popular Vector Databases
GitHub stars on October 24, 2026, from the GitHub API:
| Project | Stars |
|---|---|
| Milvus | 46,468 |
| Qdrant | 35,227 |
| Chroma | 29,569 |
| pgvector | 23,466 |
| Weaviate | 16,907 |
| LanceDB | 11,762 |
| sqlite-vec | 8,211 |
General-purpose engines are not vector-specific: Elasticsearch 78,365, Redis 77,086, DuckDB 42,497, OpenSearch 13,902. Pinecone, turbopuffer, Atlas and Vectorize are closed source. Popularity is not fit.
Fastest Vector Database? What Benchmarks Say and What I Measured
What Public Benchmarks Say
ANN-Benchmarks compares nearest-neighbour algorithms and libraries on recall against queries per second. VectorDBBench runs insertion, search, filtered search and full-text workloads up to 100M vectors, but Zilliz, a vendor, publishes it. Its cases go up to 100M vectors, far more than a docs site has.
What I Measured
I used 2,159 real chunks from the FastAPI docs (151 pages crawled, 147 produced chunks). The docs are MIT licensed, robots.txt has no disallow rules, and I crawled them with WebCrawlerAPI. Chunks split on headings, max 350 tokens, embedded with text-embedding-3-small (1,536 dims). Six local stores on an Apple M2 Max with 32 GB RAM, 200 queries x 5 passes, k=10, exact scan vs indexed where the store offers both. The filter is section = 'advanced' (18.9% of chunks). Recall is against exact cosine search in numpy. Crawl cost $0.306, embeddings $0.011.
Versions: pgvector 0.8.7 (Postgres 17.11), Qdrant 1.20.0, Chroma 1.5.9, LanceDB 0.40.0, sqlite-vec 0.1.9, DuckDB 2.0.0.

Results
| Store (mode) | recall@10 (plain / filtered) | p50 / p95 ms | p50 / p95 ms filtered | insert + index s | size MB |
|---|---|---|---|---|---|
| pgvector (exact) | 1.000 / 1.000 | 6.94 / 15.34 | 2.82 / 5.71 | 1.86 + 0.03 | 18.3 |
| pgvector (HNSW) | 0.999 / 1.000 | 2.08 / 3.61 | 2.54 / 4.16 | 1.21 + 1.22 | 36.0 |
| Qdrant (exact) | 1.000 / 1.000 | 2.89 / 4.54 | 2.84 / 4.76 | 1.78 + 0.01 | 24.3 |
| Qdrant (HNSW) | 1.000 / 1.000 | 3.06 / 4.82 | 3.20 / 4.92 | 1.22 + 0.42 | 38.7 |
| Chroma (HNSW) | 1.000 / 1.000 | 1.66 / 2.03 | 4.16 / 5.04 | 0.78 | 21.5 |
| LanceDB (exact) | 1.000 / 1.000 | 4.22 / 5.52 | 5.08 / 6.33 | 0.03 + 0.00 | 13.3 |
| LanceDB (IVF_HNSW_SQ) | 0.992 / 0.997 | 1.51 / 2.27 | 1.68 / 2.11 | 0.03 + 0.12 | 17.0 |
| sqlite-vec (exact) | 1.000 / 1.000 | 13.33 / 21.58 | 12.97 / 21.56 | 0.52 | 19.1 |
| DuckDB vss (exact) | 1.000 / 1.000 | 7.42 / 8.35 | 3.73 / 4.59 | 3.77 + 0.00 | 11.8 |
| DuckDB vss (HNSW) | 1.000 / 1.000 | 2.76 / 3.47 | 3.96 / 4.86 | 3.98 + 0.43 | 25.4 |



At this size, every store answered in single-digit to low double-digit milliseconds, and indexed recall was 0.992 or higher. What differs is size on disk (11.8 to 38.7 MB against 13.3 MB of raw vectors), ingest and setup. Qdrant and pgvector need a running server. The others are a file or a library.
Caveats. Latency is noisy on a single machine. pgvector and Qdrant were measured over loopback servers, the other four in-process, so the numbers are not apples to apples. I lowered Qdrant's indexing_threshold to force HNSW, because at this size its default collection is a plain scan. pgvector's filtered query used a btree on section plus an exact sort, not the HNSW index, so its filtered recall is exact by construction. Queries are heading_path strings, which test index behaviour, not answer quality. The numbers are one run on one machine. For a larger comparison, the pgvector tutorial has a 100k-row synthetic run.
What This Means for a Docs Knowledge Base
Latency rarely decides it at this size. Filters, ops and price do.
Pick by Use Case

- Support bot over your docs: pgvector, otherwise Qdrant or Pinecone. Add hybrid search early, since users paste error codes.
- In-product help search: your app database. With Postgres, permissions and content stay in one query.
- Internal wiki: Qdrant or Weaviate self-hosted, if data cannot leave your network.
- Multi-tenant SaaS: one index and a tenant_id filter first. One index per customer only for big or isolated tenants.
- Agent memory: LanceDB, Chroma or sqlite-vec locally, pgvector on a server.
The Part Other Comparisons Skip: Getting Web Content In
Garbage in, garbage out, whatever database you pick. Raw HTML brings menus, cookie banners and footers, and they show up as top matches for unrelated questions. The input should be clean markdown, one file per page. Building that yourself is covered in HTML to clean markdown in JavaScript.

The pipeline is the same for every database: crawl, chunk, embed, upsert with the source URL as metadata. WebCrawlerAPI does the crawl step and returns markdown per page, so JavaScript rendering and retries are not your problem. A complete Python version of this pipeline, with no database, is in the tutorial How to Build a Knowledge Base for AI From a Website. The vector store part is a placeholder, since it differs per database:
// Node 18+ (ES modules). Install: npm i webcrawlerapi-js
import webcrawlerapi from "webcrawlerapi-js";
const client = new webcrawlerapi.WebcrawlerClient(process.env.WEBCRAWLERAPI_KEY);
// Placeholder: chunk the markdown, embed the chunks, then
// delete the old chunks for this URL and insert the new ones.
async function embedAndUpsert(url, markdown) {
// your vector store code goes here
}
// Crawl the docs and wait for the job to finish
const result = await client.crawl({
url: "https://docs.example.com/",
output_formats: ["markdown"],
items_limit: 500,
// stay inside the docs, skip the marketing site
whitelist_regexp: "^https://docs\\.example\\.com/.*",
});
for (const item of result.job_items) {
const markdown = await item.getContent(); // markdown for this page
if (!markdown) continue; // failed or empty page, skip it
await embedAndUpsert(item.original_url, markdown);
}
Try the crawl-to-markdown step on your own docs with the crawl API (pricing on the crawl API page), or read the JS SDK docs. For bigger sites, crawlAsync plus webhook_url avoids waiting on a long job. The whole flow, with measured numbers, is in website to markdown for knowledge bases. Crawl-to-files and chunking tutorials are coming.
Freshness is the second half. A stale chunk gives an old answer with full confidence. Re-crawl on a schedule, hash each page's markdown, and for changed pages delete the old chunks by URL and upsert new ones. Delete chunks for pages that disappeared. In my pgvector run, 19 of 20 pages were skipped on re-crawl because their hash was unchanged.
Cost and Migration Notes
A 5,000-page site is about 50,000 chunks and 308 MB of raw vectors. Several free tiers hold that on paper (Pinecone Starter 2 GB, Zilliz 5 GB, Weaviate up to 100,000 objects), but I did not load 50,000 chunks into each service. Free tiers also limit regions, read units and support.
On lock-in: keep your source markdown and content hashes, next to the vectors or in object storage. Switching vector databases or embedding models then means re-upserting from files, with no re-crawl.
Pinecone Alternatives
- pgvector: if you run Postgres, one system instead of two.
- Qdrant: open source, the same engine locally, self-hosted and managed.
- turbopuffer: managed, usage-based, no free tier.
- Weaviate: hybrid search, free 100,000-object cloud tier.
- Chroma: runs in-process or as a server in Docker, for small apps.
FAQ
What is the best vector database for a knowledge base?
The one that adds the least work to your stack. Already on Postgres: pgvector. No ops wanted: Pinecone. Free self-hosting with a managed option: Qdrant. A docs site of a few thousand pages is tens of thousands of chunks, so almost any store works. See the pick-by-situation list at the top of this post.
What is the cheapest vector database?
Your existing Postgres with pgvector, or a SQLite or LanceDB file, costs nothing extra. Among managed services, as of November 11, 2026: Zilliz Serverless from $0, Chroma Cloud $0 plus usage, Redis Essentials about $5 per month, Cloudflare Vectorize $5 per month (Workers Paid), MongoDB Atlas Flex $8 to $30 per month, turbopuffer $16 per month minimum, Pinecone Builder $20 per month. Limits differ, see the pricing table.
Is there a free vector database?
Yes. pgvector, Qdrant, Chroma, Milvus, LanceDB, sqlite-vec and DuckDB are free to self-host. Free cloud tiers exist at Pinecone (2 GB), Qdrant (1 GB RAM, 4 GB disk), Weaviate (100,000 objects), Zilliz (5 GB), Chroma, Supabase, Neon, MongoDB Atlas and Redis, each with limits. For a side project, Pinecone Starter, the Zilliz free tier or a local sqlite-vec file is enough.
What is the best self-hosted vector database?
pgvector if you already run Postgres, since it adds no new service. Otherwise Qdrant: one Docker container, Apache-2.0, payload filters and hybrid search, with a managed cloud if you stop wanting to run it. Weaviate is the pick when hybrid search matters most, Milvus when you have hundreds of millions of vectors. Setup steps are in the Qdrant Docker tutorial.
What are the most popular vector databases?
By GitHub stars on October 24, 2026: Milvus 46,468, Qdrant 35,227, Chroma 29,569, pgvector 23,466, Weaviate 16,907, LanceDB 11,762. Pinecone is closed source and has no star count. Popularity says little about fit for a small docs knowledge base, so pick on ops burden, filters and price.
What is the best local vector database?
For a docs knowledge base on one machine, sqlite-vec, LanceDB, Chroma and DuckDB all work without a server. In my test on 2,159 chunks their stores took 11.8 to 25.4 MB on disk. sqlite-vec is brute force only, and DuckDB's index persistence is still experimental. See the local and embedded section.
What is the best hosted or managed vector database service?
Pinecone for zero ops, Zilliz Cloud for very large scale, Qdrant Cloud or Weaviate Cloud for an open source engine with a managed option, turbopuffer for usage-based pricing, and Supabase if you want pgvector without running Postgres. Compare their free tiers and minimums in the pricing table.
What is the fastest vector database?
At knowledge base size it is the wrong question. On 2,159 real chunks, six local stores answered in about 1.5 to 13 ms at p50, so disk size, ingest time and setup differ more than latency. VectorDBBench runs cases up to 100M vectors, far more than a docs site has. I name no winner.
What are the best open source vector databases?
Qdrant, Milvus, Weaviate, Chroma, pgvector and LanceDB are the main ones. Qdrant is free to self-host and has a managed cloud. Milvus fits very large scale, and Milvus Lite runs as a local folder for prototypes. pgvector fits teams already on Postgres. LanceDB runs inside your app. Check each licence, as Weaviate and Elasticsearch have mixed terms.
What are good Pinecone alternatives?
pgvector if you run Postgres, Qdrant for open source with a cloud option, turbopuffer for usage-based pricing, Weaviate for hybrid search and Chroma for small apps. See the Pinecone alternatives section for one line on each. Each one trades some ops work for a different price model.
Do I need a vector database at all?
Not always. A few hundred to a few thousand pages with mostly exact terms (error codes, API names) can be served by keyword search such as Postgres full-text or SQLite FTS5. Add vectors when users phrase questions differently from the docs. pgvector vs Pinecone is covered in the pgvector tutorial.
Conclusion
The best vector database for a docs knowledge base is the one that adds the least work to your stack. On Postgres, pgvector. No ops, Pinecone. Self-hosted dedicated, Qdrant. Local, sqlite-vec or LanceDB.
What usually breaks a knowledge base is dirty input and stale pages. Spend time there: clean markdown in, hashes for change detection, delete by URL on update. To start, follow website to markdown for knowledge bases, then the pgvector tutorial. If you are still choosing a crawler, I compared the options in best web crawler API.
