How to Build a Production RAG Application: Ingestion, Chunking, Embeddings, and Reranking

1. The Anatomy of Production-Grade RAG

Building a proof-of-concept RAG application with 10 lines of LangChain code takes an afternoon; engineering a production RAG application capable of sub-100ms retrieval, multi-format document ingestion, and 0% critical hallucinations requires a disciplined five-stage pipeline.

Production RAG involves: (1) Ingestion & Semantic Chunking, (2) Vector Embedding & Metadata Indexing, (3) Hybrid Retrieval (Dense Vector + BM25 Lexical Keyword), (4) Cross-Encoder Reranking, and (5) Citation-Constrained Synthesis with Semantic Caching.

2. Stage 1: Document Chunking Strategies

Naive fixed-character chunking (e.g. splitting every 500 characters) tears sentences apart and destroys contextual meaning. Choose your chunking strategy based on document structure:

3. Stage 2 & 3: Hybrid Search & Cross-Encoder Reranking

Pure vector search (dense embeddings) struggles with exact alphanumeric strings, such as product SKUs, error codes, and customer names. Production systems combine Dense Vector Search with Sparse BM25 Lexical Matching (Reciprocal Rank Fusion) and pass candidates through a Cross-Encoder Reranker (Cohere Rerank or BGE-Reranker-v2).

// lib/rag/hybrid-query.ts
import { Pinecone } from "@pinecone-database/pinecone";
import { CohereClient } from "cohere-ai";

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const cohere = new CohereClient({ token: process.env.COHERE_API_KEY! });

export async function executeHybridRagQuery({
  query,
  tenantId,
  topK = 5,
}: {
  query: string;
  tenantId: string;
  topK?: number;
}) {
  const index = pinecone.index("enterprise-knowledge");

  // 1. Generate Query Vector Embedding
  const embedding = await generateEmbedding(query);

  // 2. Perform Tenant-Isolated Vector Search with Metadata Filter
  const vectorResults = await index.query({
    vector: embedding,
    topK: topK * 3, // Retrieve candidate pool
    filter: { tenantId: { $eq: tenantId } },
    includeMetadata: true,
  });

  const candidateDocuments = vectorResults.matches.map((m) => ({
    id: m.id,
    text: String(m.metadata?.text || ""),
    source: String(m.metadata?.sourceUrl || ""),
  }));

  // 3. Cross-Encoder Reranking for Precision
  const reranked = await cohere.rerank({
    model: "rerank-v3.5",
    query: query,
    documents: candidateDocuments.map((d) => d.text),
    topN: topK,
  });

  // 4. Return Top Grounded Chunks with Exact Provenance
  return reranked.results.map((r) => ({
    ...candidateDocuments[r.index],
    relevanceScore: r.relevanceScore,
  }));
}

4. Stage 4 & 5: Citation Constraints & Token Margin Protection

To eliminate hallucinations, inject strict system instructions: 'Answer the question strictly using the provided Context. For every claim made, cite the [Source ID]. If the Context does not provide the answer, respond with: I do not have enough information to answer based on the provided documents.'

Furthermore, implement a Redis semantic caching layer: when a new user query has >0.96 cosine similarity to a recently answered query, return the cached answer instantly at sub-10ms latency with 0 token spend.