How do you keep a crawled knowledge base in sync as the source changes?

Your first crawl captures a snapshot of a documentation site, product guide, or internal wiki — but source content changes daily. A single crawl becomes stale, and your knowledge base diverges from the live source. The question then shifts from "how do I crawl once?" to "how do I keep it fresh without burning through budget crawling everything every hour?"

The answer lies in detecting what actually changed and recrawling only that, leaving the rest alone.

The naive approach

Re-crawl the entire site every time you think something might have changed:

import requests
from datetime import datetime

def recrawl_everything(urls):
    """Crawl all URLs every scheduled interval"""
    results = []
    for url in urls:
        response = requests.get(url)
        results.append({
            "url": url,
            "content": response.text,
            "crawled_at": datetime.now()
        })
    return results

# Run this hourly. 100 URLs × 24 hours = 2,400 crawls/day.
# Cost scales linearly with crawl frequency.

This works but wastes API credits and time re-parsing unchanged content. A documentation page that hasn't changed shouldn't trigger re-embedding or re-indexing of the same text.

Change detection: let the server tell you

Use HTTP headers that pages already send — they're free signals:

import requests
from datetime import datetime

def check_if_changed(url, last_etag=None, last_modified=None):
    """Check if page changed using ETag or Last-Modified headers"""
    response = requests.head(url)  # HEAD is cheaper than GET
    
    current_etag = response.headers.get("etag")
    current_modified = response.headers.get("last-modified")
    
    # ETag changed = content changed
    if current_etag and current_etag != last_etag:
        return True, current_etag, current_modified
    
    # Last-Modified is newer than our copy
    if current_modified and current_modified != last_modified:
        return True, current_etag, current_modified
    
    return False, current_etag, current_modified

# Check all 100 URLs daily. HEAD requests are fast and cheap.
# Only re-crawl (GET) the ~10 that actually changed.

If ETag or Last-Modified headers are absent (they often are), hash the content yourself:

import hashlib

def content_hash(text):
    return hashlib.sha256(text.encode()).hexdigest()

# Store hash of last crawled content
last_hash = "a1b2c3d4..."

# On next crawl, compare
new_response = requests.get(url)
new_hash = content_hash(new_response.text)

if new_hash != last_hash:
    # Content changed — update knowledge base entry
    update_knowledge_base(url, new_response.text)
else:
    # Content unchanged — skip re-embedding
    print(f"Skipped {url}: no changes")

Incremental vs full recrawl: cost tradeoffs

Incremental (change detection):

• Run HEAD checks on all URLs weekly
• Full GET only on changed pages
• Cost: ~1% of full-recrawl budget for similar coverage
• Trade-off: Misses structural changes that don't alter content (link changes, navigation)

Full recrawl:

• Re-GET every page on a fixed schedule
• Catches everything but bleeds budget
• Cost: 100% of budget, predictable
• Trade-off: Simple to implement, expensive at scale

Hybrid (recommended):

• Weekly change detection (HEAD + hash)
• Monthly full recrawl as a safety net
• Typical cost: 15-20% of naive hourly budget for similar freshness

Handling deleted pages

When a source page is deleted (404 response), don't leave stale knowledge base entries:

def sync_knowledge_base(checked_urls, kb_entries):
    """Remove KB entries for pages that no longer exist"""
    for url in kb_entries:
        if url not in checked_urls:
            continue
        
        response = requests.head(url)
        if response.status_code == 404:
            delete_from_knowledge_base(url)
            print(f"Removed {url}: source deleted")

Re-embedding only changed chunks

When content changes, re-hash individual sections (paragraphs, code blocks) instead of the whole page:

def chunk_content(html):
    """Split content into semantic chunks"""
    # Split by heading or paragraph boundaries
    chunks = []
    for elem in parse_sections(html):
        chunk_hash = content_hash(elem.text)
        chunks.append({"text": elem.text, "hash": chunk_hash})
    return chunks

# On update, compare chunk hashes
new_chunks = chunk_content(new_html)
old_chunks = chunk_content(old_html)

chunks_to_reembed = [
    new_chunks[i] for i in range(len(new_chunks))
    if i >= len(old_chunks) or new_chunks[i]["hash"] != old_chunks[i]["hash"]
]

# Reembed only the changed chunks, leave the rest alone

This cuts re-embedding costs by 60–80% when edits are localized.

Tips

• Prefer ETag over Last-Modified — it catches content changes even if the timestamp hasn't updated.
• HEAD requests are 2–5x cheaper than GET; use them for polling.
• Set a reasonable crawl floor: checking too frequently defeats the cost savings; daily checks are usually sufficient.
• Store hashes or ETags alongside your knowledge base entries so you can compare on next run.
• See how often should you crawl for scheduling strategies.

WebCrawlerAPI supports change detection headers and incremental crawling out of the box — you define a recrawl schedule and we track what changed, so you only re-embed when it matters.