Building a RAG Chatbot with LangChain and FAISS

When I first started exploring Generative AI, the biggest blocker was getting an LLM to answer questions about my own data. That's where Retrieval-Augmented Generation (RAG) comes in.

RAG is elegant in its simplicity: instead of retraining a model, you retrieve the relevant context at query time and hand it to the LLM along with the question. This gives you grounded, up-to-date answers with sources.

How RAG works

1. Chunk — split documents into smaller pieces
2. Embed — convert each chunk into a vector
3. Index — store vectors in a vector database
4. Retrieve — at query time, find the most similar chunks
5. Generate — let the LLM answer using only the retrieved context

Here's the core of what I built:

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings
 
# 1 & 2. Load and chunk
loader = PyPDFLoader("docs/product.pdf")
chunks = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50
).split_documents(loader.load())
 
# 3. Embed and index
index = FAISS.from_documents(
    chunks, HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
)
 
# 4. Retrieve
retriever = index.as_retriever(search_kwargs={"k": 4})
relevant = retriever.invoke("What are the pricing tiers?")

The tricky part: chunking strategy

Chunking is where quality lives or dies. Too small and you lose context; too large and you dilute relevance. A few rules I've landed on:

• Use RecursiveCharacterTextSplitter with separators that respect document structure
• Keep chunk sizes between 300–800 tokens
• Overlap by 10–15% so context isn't split mid-sentence
• Consider semantic chunking for dense technical documents

Grounding the answer

The last step matters most: the LLM should answer only from retrieved context, and admit when it doesn't know. This prevents hallucination:

from langchain.chains import RetrievalQA
from langchain_community.llms import HuggingFaceHub
 
qa = RetrievalQA.from_chain_type(
    llm=HuggingFaceHub(repo_id="mistralai/Mistral-7B-Instruct-v0.2"),
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True,
)
 
result = qa.invoke("What are the pricing tiers?")
print(result["result"])

Key takeaways

• RAG is the fastest way to get an LLM working with your private data
• FAISS is a great starting point — lightweight, local, no infra needed
• Chunking quality directly drives answer quality
• Always return sources so users can verify

This project taught me the full RAG pipeline end to end, and it became the foundation for a few of the SaaS ideas I'm exploring now.