RAG Architect

A faithfulness score of 0.61 means 39% of your generated answers aren't grounded in retrieved context — the model is filling gaps with plausible-sounding information. Here's the systematic fix.

Step 1 — Diagnose whether it's a retrieval problem or a generation problem:

# Test: does the right chunk come back at all?
results = retriever.search("what port does the auth service use")
for r in results:
    print(r['text'][:200])
# If 'auth service' + port number appears in top-3 results: generation problem
# If it doesn't appear: retrieval problem

If it's a retrieval problem — your chunks are too large:

Port numbers in a 400-token chunk get diluted. The embedding represents the full chunk, not the specific fact.

Fix: smaller chunks for configuration docs (100-150 tokens), with parent document retrieval:

# Small chunks for retrieval precision
chunk = "Auth service configuration: HOST=0.0.0.0, PORT=8443, TLS=true"
# Large parent document for context in generation
parent = full_service_config_page

If it's a generation problem — your system prompt is too permissive:

# Current (too loose):
system = "Answer questions about our infrastructure using the context provided."

# Better (grounding constraint):
system = """Answer ONLY based on the provided context. 
If the context does not contain the answer, say "I don't have that information in the docs" — 
do NOT guess or use your training knowledge.
Always cite the specific document and section your answer comes from."""

Step 2 — Add source attribution to expose hallucinations:

# Force the model to cite sources — citation failures reveal retrieval gaps
prompt = f"""Context:\n{context}\n\nQuestion: {query}\n
Answer the question and end with: Source: [exact document title and section]"""

Step 3 — Evaluate per-query with RAGAS:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_recall])
# Target: faithfulness > 0.90, context_recall > 0.85

Address retrieval first — getting faithfulness to 0.90+ usually requires both smaller chunks and a stricter system prompt.

Your constraints — multi-tenancy with data isolation, existing Postgres on RDS, and 500k docs × 200 customers (100M vectors at max scale) — point strongly toward a specific decision.

Recommendation: Start with pgvector, plan for Pinecone at scale

Phase 1 (0–200 customers): pgvector on RDS

You already have Postgres. Add the extension:

CREATE EXTENSION vector;

-- Per-tenant isolation via tenant_id column (NOT separate tables)
CREATE TABLE embeddings (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL REFERENCES tenants(id),
  document_id UUID NOT NULL,
  chunk_text TEXT,
  embedding vector(1536),
  metadata JSONB,
  created_at TIMESTAMPTZ DEFAULT now()
);

-- HNSW index per-tenant for fast ANN search
CREATE INDEX CONCURRENTLY embeddings_tenant_embedding_hnsw
  ON embeddings USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

-- Row-level security enforces tenant isolation at DB layer
ALTER TABLE embeddings ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON embeddings
  USING (tenant_id = current_setting('app.current_tenant_id')::UUID);

At 500k docs × 1536 dims × 4 bytes = ~3GB per tenant. 200 tenants = ~600GB. RDS db.r7g.2xlarge (64GB RAM) handles this if you set shared_buffers = 16GB and use HNSW (not ivfflat).

Phase 2 (>500 customers): Evaluate Pinecone namespaces

Pinecone's namespace feature maps 1:1 to tenants, scales to billions of vectors, and handles the operational burden of indexing. At 2,000 customers: $70/namespace × 2,000 = $140k/month — compare to RDS scaling costs.

Avoid Weaviate here: Its multi-tenancy model requires per-tenant collections, which becomes operationally complex at 2,000 tenants.

The pgvector start is low-risk: your schema, queries, and application code transfer directly to pgvector-compatible hosted services (Neon, Supabase, or Aurora) if you need to migrate later.