RAG Explained: Build a Production AI Knowledge Base from Scratch (2026)

What Is RAG and Why Does It Matter?

Large language models like Claude and GPT-4o are extraordinarily capable — but they have two critical limitations for real-world business applications. They don't know about your private data. And their knowledge has a training cutoff. Ask Claude about your company's internal documentation, your product's support articles, or this morning's news — it can't help you without more context.

Retrieval-Augmented Generation (RAG) solves both problems. Instead of relying purely on what the model was trained on, RAG retrieves relevant information from your own data sources at query time and passes that information as context to the model. The model then generates an answer grounded in your specific, up-to-date knowledge — citing sources, not hallucinating facts.

The practical impact is enormous: customer support bots that actually know your product, internal search that understands natural language questions about your codebase, research tools that synthesize insights from hundreds of documents, and AI assistants that have genuine expertise in your specific domain.

Fine-tuning teaches a model new skills and styles. RAG gives a model new knowledge. If you want Claude to know about your product's documentation and support tickets — use RAG. If you want Claude to always respond in a specific format or writing style — consider fine-tuning. For most business applications, RAG is the right choice : it's faster, cheaper, updatable, and explainable.

How RAG Works: The Complete Picture

RAG has two distinct phases that are easy to confuse but essential to understand separately. The indexing phase happens once (or on a schedule when data changes). The query phase happens on every user request. Getting both phases right determines the quality and latency of your RAG system.

Chunking: The Decision That Makes or Breaks RAG Quality

Chunking is splitting your documents into pieces before embedding them. This is the most underestimated decision in RAG system design. Chunk too large and your retrieved context is unfocused, wasting tokens and diluting relevance. Chunk too small and each chunk lacks enough context for the LLM to understand it.

The right chunking strategy depends entirely on your content type. Here are the four main approaches and when to use each.

The Golden Rule of Chunk Size

For most general-purpose RAG applications: 512 tokens chunk size with 50–100 token overlap is a solid starting point. Here's the practical reasoning:

512 tokens is enough to capture a complete idea or explanation without being too broad
The overlap prevents losing context that spans chunk boundaries
At this size, you can fit 8–10 retrieved chunks in a Claude Sonnet context window alongside a detailed system prompt
Smaller models or cost-sensitive applications can use 256 tokens; longer documents with dense information benefit from 1024

import  {  encode ,  decode  }  from   'gpt-tokenizer' 

 interface   ChunkOptions  {
  chunkSize?:  number        // tokens per chunk (default: 512) 
  chunkOverlap?:  number     // overlap tokens (default: 64) 
  separators?:  string []   // try these in order 
}

 interface   Chunk  {
  content:  string 
  tokenCount:  number 
  index:  number 
  metadata?:  Record < string ,  unknown >
}

 export function   chunkText (
  text:  string ,
  options:  ChunkOptions  = {}
):  Chunk [] {
   const  {
    chunkSize =  512 ,
    chunkOverlap =  64 ,
    separators = [ '\n\n' ,  '\n' ,  '. ' ,  ' ' ,  '' ]
  } = options

   const  chunks:  Chunk [] = []
   const  tokens =  encode (text)

   let  start =  0 
   let  chunkIndex =  0 

   while  (start  <  tokens.length) {
     const  end = Math. min (start + chunkSize, tokens.length)
     const  chunkTokens = tokens. slice (start, end)
     const  content =  decode (chunkTokens)

     // Find the last clean separator within the chunk 
     let  cleanContent = content
     if  (end  <  tokens.length) {
       for  ( const  sep  of  separators) {
         const  lastSep = content. lastIndexOf (sep)
         if  (lastSep > content.length *  0.5 ) {
          cleanContent = content. slice ( 0 , lastSep + sep.length)
           break 
        }
      }
    }

    chunks. push ({
      content: cleanContent. trim (),
      tokenCount:  encode (cleanContent).length,
      index: chunkIndex++,
    })

     // Next chunk starts with overlap 
     const  overlapStart = Math. max ( 0 , end - chunkOverlap)
    start = overlapStart
     if  (start >= tokens.length)  break 
  }

   return  chunks. filter (c => c.content.length >  0 )
}

Embeddings: Choosing the Right Model

An embedding model converts text into a dense numerical vector — a list of 768 to 3,072 numbers — that captures the semantic meaning of the text. Similar meaning = vectors that point in similar directions in high-dimensional space. This is how "What is the refund policy?" and "How do I get my money back?" map to nearby vectors, even though they share no words.

You must use the same embedding model for indexing and querying. If you index documents with text-embedding-3-small , you must embed user queries with text-embedding-3-small . The vector spaces are model-specific — mixing models produces meaningless similarity scores. This also means migrating embedding models requires re-indexing all your documents.

import  OpenAI  from   'openai' 

 const  openai =  new   OpenAI ()

 const  EMBEDDING_MODEL =  'text-embedding-3-small' 
 const  BATCH_SIZE =  100    // OpenAI allows up to 2048 inputs per request 

 export async function   embedTexts (
  texts:  string []
):  Promise < number [][]> {
   const  allEmbeddings:  number [][] = []

   // Process in batches to respect rate limits 
   for  ( let  i =  0 ; i  <  texts.length; i += BATCH_SIZE) {
     const  batch = texts. slice (i, i + BATCH_SIZE)

     const  response =  await  openai.embeddings. create ({
      model: EMBEDDING_MODEL,
      input: batch,
      encoding_format:  'float' ,
    })

     // Preserve order — API returns in same order as input 
     const  batchEmbeddings = response.data
      . sort ((a, b) => a.index - b.index)
      . map (d => d.embedding)

    allEmbeddings. push (...batchEmbeddings)

     // Gentle rate limit buffer between batches 
     if  (i + BATCH_SIZE  <  texts.length) {
       await new   Promise (r =>  setTimeout (r,  200 ))
    }
  }

   return  allEmbeddings
}

 export async function   embedQuery (query:  string ):  Promise < number []> {
   const  [embedding] =  await   embedTexts ([query])
   return  embedding
}

Vector Databases: pgvector vs Pinecone vs Qdrant

The vector database stores your embeddings and enables fast similarity search. Choosing the right one depends on your scale, budget, and operational preferences. Here's the honest comparison.

The Vector DB Decision in One Paragraph

Already using Supabase? Use pgvector — it's free, it's in your existing database, your metadata queries use familiar SQL, and it handles millions of vectors with a proper HNSW index. Need <10ms p99 latency at scale? Use Qdrant managed. Want the simplest API-first experience? Use Pinecone, but budget for costs at scale. For 90% of RAG applications under 5M vectors, pgvector is the right default choice.

Setting Up pgvector with Supabase

-- Enable pgvector extension 
 create extension if not exists  vector;

 -- Documents table (source files) 
 create table   documents  (
  id uuid  default  gen_random_uuid()  primary key ,
  title text  not null ,
  source_url text,
  file_type text,                     -- 'pdf' | 'markdown' | 'webpage' 
  created_at timestamptz  default  now(),
  updated_at timestamptz  default  now()
);

 -- Chunks table (embedded text segments) 
 create table   document_chunks  (
  id uuid  default  gen_random_uuid()  primary key ,
  document_id uuid  references  documents(id)  on delete cascade ,
  content text  not null ,
  embedding vector( 1536 ),            -- 1536 for text-embedding-3-small 
  token_count integer,
  chunk_index integer,
  metadata jsonb  default   '{}' ,       -- page number, section, etc. 
  created_at timestamptz  default  now()
);

 -- HNSW index for fast approximate nearest neighbor search
-- m=16, ef_construction=64 are good defaults for most use cases 
 create index   document_chunks_embedding_idx 
   on  document_chunks
   using   hnsw  (embedding  vector_cosine_ops )
   with  (m =  16 , ef_construction =  64 );

 -- Optional: filter by document for namespace-like scoping 
 create index   document_chunks_document_id_idx 
   on  document_chunks (document_id);

 -- Similarity search function (used by the query pipeline) 
 create or replace function   match_chunks (
  query_embedding vector( 1536 ),
  match_threshold  float   default   0.78 ,
  match_count  int   default   8 ,
  filter_document_id uuid  default null 
)
 returns table  (
  id uuid, content text, document_id uuid,
  metadata jsonb, similarity  float 
)
 language  sql  stable 
 as  $$
   select 
    dc.id, dc.content, dc.document_id, dc.metadata,
     1  - (dc.embedding  < #>  query_embedding)  as  similarity
   from  document_chunks dc
   where 
     1  - (dc.embedding  < #>  query_embedding) > match_threshold
     and  (filter_document_id  is null or  dc.document_id = filter_document_id)
   order by  dc.embedding  < #>  query_embedding
   limit  match_count;
$$;

The Complete RAG Pipeline: Indexing to Answer

Step 1 — The Indexing Service

import  {  createClient  }  from   '@/lib/supabase/server' 
 import  {  chunkText  }  from   '@/lib/chunker' 
 import  {  embedTexts  }  from   '@/lib/embeddings' 

 interface   IndexDocumentOptions  {
  title:  string 
  content:  string 
  sourceUrl?:  string 
  fileType?:  string 
  metadata?:  Record < string ,  unknown >
}

 export async function   indexDocument (
  opts:  IndexDocumentOptions 
):  Promise <{ documentId:  string ; chunkCount:  number  }> {
   const  supabase =  createClient ()

   // 1. Insert document record 
   const  { data: doc, error: docError } =  await  supabase
    .from( 'documents' )
    .insert({
      title: opts.title,
      source_url: opts.sourceUrl,
      file_type: opts.fileType ??  'text' ,
    })
    .select( 'id' )
    .single()

   if  (docError)  throw  docError

   // 2. Chunk the content 
   const  chunks =  chunkText (opts.content, {
    chunkSize:  512 ,
    chunkOverlap:  64 ,
  })

   if  (chunks.length ===  0 )  return  { documentId: doc.id, chunkCount:  0  }

   // 3. Generate embeddings in batch 
   const  embeddings =  await   embedTexts (chunks. map (c => c.content))

   // 4. Insert chunks with their embeddings 
   const  chunkRows = chunks. map ((chunk, i) => ({
    document_id: doc.id,
    content: chunk.content,
    embedding:  `[${embeddings[i].join(',')}]` ,
    token_count: chunk.tokenCount,
    chunk_index: chunk.index,
    metadata: { ...opts.metadata },
  }))

   const  { error: chunkError } =  await  supabase
    .from( 'document_chunks' )
    .insert(chunkRows)

   if  (chunkError)  throw  chunkError

  console. log ( `Indexed "${opts.title}": ${chunks.length} chunks` )
   return  { documentId: doc.id, chunkCount: chunks.length }
}

Step 2 — The Query Engine

import  Anthropic  from   '@anthropic-ai/sdk' 
 import  {  createClient  }  from   '@/lib/supabase/server' 
 import  {  embedQuery  }  from   '@/lib/embeddings' 

 const  anthropic =  new   Anthropic ()

 interface   RAGResponse  {
  answer:  string 
  sources:  Array <{ content:  string ; similarity:  number ; metadata:  object  }>
  tokensUsed:  number 
}

 export async function   ragQuery (
  question:  string ,
  opts: { maxChunks?:  number ; threshold?:  number ; documentId?:  string  } = {}
):  Promise < RAGResponse > {
   const  { maxChunks =  8 , threshold =  0.75  } = opts
   const  supabase =  createClient ()

   // 1. Embed the user's question 
   const  queryEmbedding =  await   embedQuery (question)

   // 2. Retrieve semantically similar chunks 
   const  { data: chunks, error } =  await  supabase
    . rpc ( 'match_chunks' , {
      query_embedding:  `[${queryEmbedding.join(',')}]` ,
      match_threshold: threshold,
      match_count: maxChunks,
      filter_document_id: opts.documentId ??  null ,
    })

   if  (error)  throw  error
   if  (!chunks || chunks.length ===  0 ) {
     return  {
      answer:  "I couldn't find relevant information to answer that question." ,
      sources: [],
      tokensUsed:  0 ,
    }
  }

   // 3. Build the context from retrieved chunks 
   const  context = chunks
    . map ((c, i) =>  `[Source ${i + 1}] (similarity: ${c.similarity.toFixed(3)})\n${c.content}` )
    . join ( '\n\n---\n\n' )

   // 4. Generate answer with Claude 
   const  response =  await  anthropic.messages. create ({
    model:  'claude-sonnet-4-20260514' ,
    max_tokens:  1024 ,
    system:  `You are a helpful AI assistant. Answer questions using ONLY
the provided context. If the context doesn't contain enough information
to answer fully, say so explicitly. Always cite which sources you used
by referring to [Source N] in your answer.` ,
    messages: [{
      role:  'user' ,
      content:  `Context:\n\n${context}\n\n---\n\nQuestion: ${question}` 
    }],
  })

   const  answer = response.content[ 0 ].type ===  'text' 
    ? response.content[ 0 ].text
    :  '' 

   return  {
    answer,
    sources: chunks. map (c => ({
      content: c.content,
      similarity: c.similarity,
      metadata: c.metadata,
    })),
    tokensUsed: response.usage.input_tokens + response.usage.output_tokens,
  }
}

Step 3 — The API Route (Next.js)

import  {  NextRequest  }  from   'next/server' 
 import  {  ragQuery  }  from   '@/lib/rag-query' 
 import  {  createClient  }  from   '@/lib/supabase/server' 

 export async function   POST (req:  NextRequest ) {
   try  {
     const  supabase =  createClient ()
     const  { data: { user } } =  await  supabase.auth. getUser ()
     if  (!user)  return   Response . json ({ error:  'Unauthorized'  }, { status:  401  })

     const  { question, documentId } =  await  req. json ()
     if  (!question?.trim()) {
       return   Response . json ({ error:  'Question is required'  }, { status:  400  })
    }

     const  result =  await   ragQuery (question, {
      maxChunks:  8 ,
      threshold:  0.75 ,
      documentId,
    })

     // Log query for analytics (user_id, question, token usage) 
     await  supabase.from( 'query_logs' ). insert ({
      user_id: user.id,
      question,
      tokens_used: result.tokensUsed,
      sources_found: result.sources.length,
    })

     return   Response . json (result)
  }  catch  (error) {
    console. error ( 'RAG query error:' , error)
     return   Response . json ({ error:  'Query failed'  }, { status:  500  })
  }
}

Retrieval Strategies: Going Beyond Simple Similarity Search

Basic cosine similarity retrieval works — but it has known failure modes. Semantic search misses exact keyword matches. A user asking about "SOC 2 Type II audit" should find that exact phrase, not just semantically related security documents. The solution is hybrid retrieval — combining vector similarity with full-text keyword search.

Hybrid Search: Vector + Keyword (BM25)

Supabase supports tsvector full-text search natively. Combining it with vector similarity using Reciprocal Rank Fusion (RRF) gives you the best of both worlds: semantic understanding for natural language queries, exact keyword matching for specific terms, codes, and proper nouns.

-- Add full-text search column 
 alter table  document_chunks
   add column  fts tsvector
   generated always as  ( to_tsvector ( 'english' , content)) stored;

 create index   document_chunks_fts_idx 
   on  document_chunks  using  gin(fts);

 -- Hybrid search function: vector + BM25 with RRF scoring 
 create or replace function   hybrid_search (
  query_text text,
  query_embedding vector( 1536 ),
  match_count  int   default   8 ,
  rrf_k  int   default   60     -- RRF constant, 60 is standard 
)
 returns table (id uuid, content text, metadata jsonb, score  float )
 language  sql  stable as 
$$
   with 
    vector_results  as  (
       select  id,
              row_number ()  over ( order by  embedding  < #>  query_embedding)  as  rank
       from  document_chunks
       order by  embedding  < #>  query_embedding
       limit  match_count *  3 
    ),
    keyword_results  as  (
       select  id,
              row_number ()  over ( order by   ts_rank_cd (fts,  plainto_tsquery ( 'english' , query_text))  desc )  as  rank
       from  document_chunks
       where  fts @@  plainto_tsquery ( 'english' , query_text)
       limit  match_count *  3 
    )
   select 
    dc.id, dc.content, dc.metadata,
     coalesce ( 1.0 /(rrf_k + vr.rank),  0 ) +
     coalesce ( 1.0 /(rrf_k + kr.rank),  0 )  as  score
   from  document_chunks dc
   left join  vector_results vr  on  dc.id = vr.id
   left join  keyword_results kr  on  dc.id = kr.id
   where  vr.id  is not null or  kr.id  is not null 
   order by  score  desc 
   limit  match_count;
$$;

Query Rewriting: Handling Vague Questions

Users ask bad questions. "Tell me about the thing from last month's update" is what a user types; "What changed in the April 2026 product update regarding billing?" is what your retriever needs. Query rewriting uses Claude to reformulate the user's question before embedding it — dramatically improving retrieval quality for conversational or ambiguous queries.

import  Anthropic  from   '@anthropic-ai/sdk' 

 const  anthropic =  new   Anthropic ()

 /**
 * Generate multiple query variations to improve retrieval recall.
 * Retrieve for each variant, deduplicate results by ID.
 */ 
 export async function   expandQuery (
  originalQuery:  string ,
  numVariants:  number  =  3 
):  Promise < string []> {
   const  response =  await  anthropic.messages. create ({
    model:  'claude-haiku-4-5-20261001' ,   // Use Haiku for speed/cost 
    max_tokens:  300 ,
    messages: [{
      role:  'user' ,
      content:  `Generate ${numVariants} alternative phrasings of this search query
for a knowledge base. Return ONLY a JSON array of strings, no other text.

Original query: "${originalQuery}"

Generate variants that:
- Use different vocabulary with the same intent
- Expand acronyms or technical terms
- Include related synonyms
- Add useful context if the query is vague` 
    }]
  })

   try  {
     const  text = response.content[ 0 ].type ===  'text' 
      ? response.content[ 0 ].text :  '[]' 
     const  variants =  JSON . parse (text)  as   string []
     return  [originalQuery, ...variants]  // Include original 
  }  catch  {
     return  [originalQuery]  // Fallback to original on parse failure 
  }
}

Going to Production: What Changes at Scale

Cost Management

RAG has two main cost drivers: embedding generation (cheap but scales with document count) and LLM generation (expensive and scales with queries). The key optimization levers:

Cache query embeddings. The same user question should not re-generate embeddings. Store question → embedding in Redis with a 24-hour TTL. Hash the question string as the cache key.
Cache RAG responses. For common questions, the retrieved chunks + answer can be cached for hours or days. Customer support bots often see the same 20 questions 80% of the time.
Use Claude Haiku for simple queries. Route simple factual questions to Haiku ($0.25 per million tokens), reserve Sonnet for complex reasoning. Implement a classifier that detects query complexity.
Limit context tokens. Don't retrieve 10 chunks if 5 are sufficient. Monitor answer quality vs chunk count and find the right threshold for your content type.

Answer Quality Monitoring

RAG systems have a known failure mode: confident wrong answers when the retrieved context is insufficient. Implement these quality signals:

Retrieval confidence score. If the highest similarity score is below 0.7, flag the answer as low confidence. Show users a warning: "I'm less certain about this answer — consider verifying with primary sources."
Source attribution. Always show which chunks the answer was derived from. Users can verify, and it builds trust.
Thumbs up/down feedback. The simplest quality signal. Log question, answer, retrieved chunks, and feedback. Use failures to improve chunking and retrieval thresholds.
Hallucination detection. After generation, run a second Claude call: "Does this answer use only information from the provided context? Return YES or NO." Flag NO cases for review.

The RAG Quality Formula

Answer quality = f(retrieval quality × context quality × prompt quality). A great LLM prompt cannot fix bad retrieval. Bad chunking cannot be overcome by better embeddings. The bottleneck moves. Start by measuring each layer separately: (1) Are you retrieving the right chunks? (2) Is the context complete enough? (3) Is the LLM generating accurate answers from good context? Fix in order.

You Have a Production RAG System — Now What?

You now have every piece to build a production-quality RAG system: chunking logic, embedding generation, pgvector storage, hybrid retrieval, query rewriting, and a streaming API endpoint. The complete code in this guide is ready to deploy.

The most common next steps are adding a chat interface (conversation history + memory), building a document management UI (upload, preview, delete), and implementing fine-grained access control (users can only query documents in their workspace). All three are natural extensions of the infrastructure built here.

RAG is not a feature. It's the infrastructure that transforms a general-purpose AI into an expert on your specific domain. Build it right once and every AI feature you add becomes dramatically more powerful.