Tag: RAG

Last refreshed: August 2026

A local vector database Claude setup — indexed with your business documents, contracts, SOPs, client notes, and invoices — gives back the operational time lost to hunting through folders. The right answer appears in seconds, without any of those documents leaving the machine.

This is the full build: architecture, tools, working code, what performs well in production, what breaks, and whether the ROI justifies the setup time.

What Problem This Solves

The problem isn’t that the documents don’t exist. It’s that finding the right one — the specific contract clause, the pricing from eight months ago, the onboarding SOP for a client — takes longer than it should, and normal search doesn’t solve it.

File search matches keywords. It doesn’t understand that “what did we agree on for payment timing” and “net 30” are the same thing. A retrieval-augmented setup solves the semantic gap: the vector database finds relevant sections by meaning, Claude synthesizes them into a direct answer.

The use cases where this setup pays for itself:

• Contract and clause lookup — “What are the payment terms in the Acme agreement?” in 4 seconds vs. 3 minutes of folder navigation
• SOP retrieval — “What’s our onboarding process for new social media clients?” surfaces the relevant runbook section directly
• Client history — “What scope did we quote [client] last spring?” retrieves the invoice or email thread
• Cross-document synthesis — “What are the termination clauses across all active client contracts?” — something no file search can do

The Architecture

The stack is ChromaDB for local vector storage, Nomic Embed for on-device embeddings via Ollama, LlamaIndex for document ingestion, and Claude Sonnet via API for the reasoning step — all files stay local, Claude only sees the retrieved chunks.

Why local for the vector database: The documents never leave the machine. Claude receives only the retrieved chunks — not the full corpus. For contracts, financial records, and internal communications, this is the right boundary.

Why Claude for reasoning and not a local model: Local models (Llama 3, Mistral) handle the retrieval step comparably. They don’t handle synthesis comparably — reading five contract sections and returning a coherent, accurate answer is where Claude’s API cost is earned.

What to Index

Start with the 50 most-referenced documents. Get the workflow running and verified before expanding to the full corpus.

File types that index well:

• Contracts and agreements (PDF, DOCX)
• Internal SOPs and runbooks (MD, DOCX)
• Client notes and meeting logs (MD, TXT)
• Invoices and financial records (PDF, CSV)
• Email threads exported from Gmail (EML, TXT)

Organize before indexing. File names and folder paths become metadata attached to each chunk. A consistent folder structure takes an hour to set up and improves retrieval quality throughout:

/business-knowledge/
  /clients/
  /contracts/
  /operations/
  /finance/
  /communications/
  /reference/

Poorly named files produce confusing retrieval results. The index is only as organized as the source files.

Building the System

Step 1: Install the stack

# Ollama for local embedding
brew install ollama
ollama pull nomic-embed-text

# Python dependencies
pip install chromadb llama-index llama-index-embeddings-ollama anthropic

Step 2: Ingest and index

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

embed_model = OllamaEmbedding(model_name="nomic-embed-text")

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("business_knowledge")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

documents = SimpleDirectoryReader("./business-knowledge", recursive=True).load_data()
index = VectorStoreIndex.from_documents(
    documents,
    embed_model=embed_model,
    vector_store=vector_store
)

print(f"Indexed {len(documents)} documents")

500 files on an M2 MacBook Pro takes approximately 20–25 minutes. The index persists to disk — this runs once, then incrementally as files change.

Step 3: Build retrieval and reasoning

import anthropic

def query_business_knowledge(question: str, top_k: int = 5) -> str:
    retriever = index.as_retriever(similarity_top_k=top_k)
    nodes = retriever.retrieve(question)

    sep = chr(10) + chr(10) + "---" + chr(10) + chr(10)
    context = sep.join([
        f"Source: {node.metadata.get('file_name', 'unknown')}{chr(10)}{node.text}"
        for node in nodes
    ])

    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        messages=[{
            "role": "user",
            "content": f"""Answer this question using only the provided business documents.
If the answer isn't in the documents, say so clearly. Always cite the source file.

Question: {question}

Documents:
{context}"""
        }]
    )

    return response.content[0].text

print(query_business_knowledge("What are the payment terms in the Acme contract?"))

Always include source attribution in the prompt. When an answer returns, the source file name makes verification fast.

What Works Well in Production

Cross-document synthesis is the capability that justifies this over standard search — querying across hundreds of files simultaneously to find patterns, compare terms, or surface a specific clause is something no file search does.

Where the system consistently delivers:

Contract and clause lookup: Specific clause retrieval across a full contract library. Synthesis across multiple contracts simultaneously (termination terms, payment terms, liability caps) returns a summary across all of them at once.

SOP and runbook retrieval: Operational questions answered directly from internal documentation. Works best when SOPs are written in complete sentences rather than bullet fragments — the retrieval quality reflects the writing quality.

Client history: Invoice amounts, quoted scopes, prior project notes. Email threads sometimes split across chunks in ways that lose context — use the result as a pointer to the source document, then verify.

Cross-document pattern finding: “What are the common liability terms across our contracts?” — synthesizes across every indexed contract in one response. No file search tool does this.

What Breaks

The index is only as current as the last re-index. The most common production failure is stale data — documents updated after the last index run return old answers.

Stale index: Build re-indexing into the workflow immediately. Schedule it weekly, or trigger it automatically when files are modified. Documents that change and don’t get re-indexed are the biggest reliability risk.

Top-k ceiling: Retrieval returns the top-k chunks (default 5). A question whose complete answer requires synthesizing 20 documents gets a partial answer. Increase top_k for broad synthesis questions — at the cost of slightly more API token usage.

Numerical calculations: The system finds financial documents reliably. It should not be trusted to calculate totals across extracted text. Use it to surface the right source documents; do the arithmetic elsewhere.

Documentation debt: The index reveals gaps in internal documentation. SOPs written in ambiguous shorthand, contracts with undefined terms, emails with unclear context — all produce lower quality retrieval. The index reflects the quality of the underlying documents.

Chunk Size

512 tokens with 50-token overlap is the right starting point for mixed document types.

Adjust based on document type:

• Contracts (dense, long): 512–768 tokens, 100-token overlap
• SOPs (structured, modular): 256–512 tokens, 50-token overlap
• Emails (short, conversational): 256 tokens, 25-token overlap
• Financial records (tabular): Parse as structured data where possible; plain text chunking loses table relationships

Metadata Filtering at Scale

Once the corpus exceeds ~200 files, adding metadata to chunks and filtering at query time significantly improves precision.

# Tag at ingestion
documents = SimpleDirectoryReader(
    "./business-knowledge",
    recursive=True,
    file_metadata=lambda filepath: {
        "document_type": filepath.split("/")[2],
        "client": filepath.split("/")[3] if len(filepath.split("/")) > 3 else "internal"
    }
).load_data()

# Filter at retrieval
retriever = index.as_retriever(
    similarity_top_k=5,
    filters={"document_type": "contracts"}
)

“What are our SOPs for [client]?” filtered to that client’s folder returns meaningfully more accurate results than querying the full corpus.

ROI

Setup takes roughly one full day. At 25 minutes saved per week on document lookups, break-even is approximately 6–8 weeks.

The less quantifiable return: operational confidence. Questions that previously required folder-hunting get answered in seconds. That reduces the cognitive overhead of running a multi-client operation and changes how quickly decisions get made.

Related on Tygart Media: how to use Claude · Anthropic API key.

Frequently Asked Questions

What to Read Next

Anthropic Console: API Keys and the Workbench

Claude AI Pricing — All Plans and API Rates

Claude API Model IDs and Strings

History of Anthropic

>Part of the complete guide: Your Laptop Is Already an AI PC