Hybrid RAG over PDF Documents

Hybrid retrieval-augmented generation for a PDF document corpus. The system chunks PDFs, builds BM25 and FAISS dense indices, fuses both rankings with Reciprocal Rank Fusion, generates grounded answers with Groq, and evaluates BM25 vs dense vs hybrid retrieval.

Credits

• Mentor: Professor Ahmet Yuksel
• Collaborator: Shaolong Lin

How The Pipeline Works

This project is a hybrid RAG system. RAG means:

Retrieve relevant document chunks -> add them to the LLM prompt -> generate an answer

The LLM is not trained on the corpus. Instead, the app searches the corpus at question time, sends only the most relevant chunks to the LLM, and asks the LLM to answer from that context.

flowchart TD
    A["PDF corpus<br/>data/papers/*.pdf"] --> B["Extract text<br/>pypdf"]
    B --> C["Split into chunks<br/>500 words, 50-word overlap"]
    C --> D["Chunk store<br/>indices/chunks.pkl"]

    D --> E["BM25 index<br/>keyword search"]
    D --> F["SentenceTransformer embeddings<br/>all-MiniLM-L6-v2"]
    F --> G["FAISS index<br/>vector search"]

    E --> H["Hybrid retrieval<br/>RRF merge"]
    G --> H

    H --> I["Top chunks"]
    I --> J["Prompt context"]
    J --> K["Groq LLM"]
    K --> L["Grounded answer<br/>with chunk citations"]

Loading

Step By Step

1. Hold the corpus

   The source documents live in data/papers/ as PDFs. The pipeline is domain-agnostic: it can work with research papers, manuals, reports, policies, notes, or any other PDF collection with extractable text. If you change the corpus topic, rebuild the chunks and indices.

2. Chunk the documents

   src.ingest extracts PDF text page by page, splits it into about 500-word chunks with 50 words of overlap, assigns each chunk an id, and stores the result in indices/chunks.pkl.

   A chunk contains the text plus metadata:

   {
     "id": 42,
     "source": "paper.pdf",
     "page": 7,
     "text": "chunk text..."
   }

3. Build two retrieval indexes

   The same chunks are indexed two ways:

   flowchart LR
       A["indices/chunks.pkl"] --> B["BM25<br/>exact keyword matching"]
       A --> C["SentenceTransformer<br/>meaning vectors"]
       C --> D["FAISS<br/>fast vector search"]
   

   Loading

   BM25 is the sparse keyword search algorithm. It is good when the user's words match the document words.

   FAISS is the dense vector search engine. It does not create embeddings itself. It stores and searches vectors created by the embedding model.

4. Embed the meaning vectors

   The embedding model is sentence-transformers/all-MiniLM-L6-v2 by default. It is an external SentenceTransformer model, not CLIP, not Groq, and not OpenAI embeddings.

   chunk text -> all-MiniLM-L6-v2 -> numeric vector -> FAISS
   user question -> all-MiniLM-L6-v2 -> numeric vector -> FAISS search
   

   The default embedding model is configured with EMBEDDING_MODEL in src/config.py. You can swap it by setting another SentenceTransformer model name before rebuilding the FAISS index.

5. Store the vectors

   The vectors are stored inside indices/faiss.index. The project does not separately save raw vectors as .npy or .json.

   Related artifacts:

   File	What it stores
   indices/chunks.pkl	Original chunk text plus id, source, and page
   indices/bm25.pkl	BM25 keyword index
   indices/faiss.index	FAISS vector index containing the dense embeddings
   indices/embeddings.json	Metadata about the embedding model, dimension, and count

6. Hybrid retrieval happens at question time

   When a user asks a question, the app runs both retrieval systems:

   question -> BM25 top 20 chunks
   question -> FAISS top 20 chunks
   BM25 + FAISS results -> RRF -> final top N chunks
   

   RRF means Reciprocal Rank Fusion. It merges the BM25 and FAISS rankings. A chunk gets a stronger final score if it ranks highly in either list, especially if both systems agree. If you see "FFS" in notes, the intended term here is RRF.

7. Send context to the LLM

   The whole corpus is not sent to Groq. Only the final retrieved chunks are placed into the prompt:

   You are a research assistant.
   Answer using ONLY the provided context passages.
   Cite sources by their [ID] inline.
   
   Context:
   [ID: 0042 | paper.pdf p.7]
   retrieved chunk text...
   
   [ID: 0118 | other-paper.pdf p.3]
   retrieved chunk text...
   
   Question: What does the corpus say about this topic?
   
   Answer:
   

   Groq runs the chat model, llama-3.3-70b-versatile by default, and returns the final grounded answer.

Short Mental Model

Corpus = library
Chunks = note cards copied from pages
BM25 = finds note cards by exact words
FAISS = finds note cards by meaning
RRF = combines both search result lists
Groq = writes the answer using only the selected note cards

Setup

Install Python 3.12 first. The default python on this machine is currently Python 3.14, which is too new for a reliable FAISS/PyTorch install path on Windows.

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
pip install -r requirements.txt

Set your Groq key before using generation or the UI:

$env:GROQ_API_KEY="your-key-here"

Optional: choose the default arXiv download topic before running src.ingest --download:

$env:ARXIV_QUERY='all:research'

Build The Corpus

Download and chunk the configured arXiv corpus:

python -m src.ingest --download --max-results 40

If arXiv is slow, place PDFs manually in data/papers/ and run:

python -m src.ingest

This writes indices/chunks.pkl.

Build Indices

python -m src.index

This writes:

• indices/bm25.pkl
• indices/faiss.index
• indices/embeddings.json

Search And Generate

Retrieve passages:

python -m src.retrieve "What are the main findings about this topic?" --top-n 5

Generate a grounded answer:

python -m src.generate "What does the corpus say about this topic?"

Run the UI:

python -m src.app

Open the printed local URL, usually http://127.0.0.1:7860.

Evaluation

Create eval/goldset.jsonl with records like:

{"qid": "q01", "question": "What evidence supports the main claim?", "answerable": true, "relevant_ids": [42, 117]}
{"qid": "q99", "question": "Which document proves an unrelated fictional claim?", "answerable": false, "relevant_ids": []}

Build it quickly by running hybrid retrieval top-20 for each question, skimming the chunks, and marking 1-3 genuinely relevant IDs. Keep 3-5 unanswerable questions for refusal checks.

Run:

python -m src.eval

The evaluator writes eval/results.json with Recall@5, MRR, and nDCG@10 for BM25, dense, and hybrid RRF.

Tests

The pure unit tests do not require FAISS, sentence-transformers, Gradio, or Groq:

python -m unittest discover -s tests -v

After installing dependencies, pytest can also run them:

pytest -q