u/Affectionate_Slip654

Great suggestion, here's what's in place so far, two unanswerable questions (stripe's per-transaction fee for iDEAL, and the maximum pci dss non-compliance fine) which the system correctly declined to answer. Each question also stores its relevant source pdfs and pages, and a checklist of key facts the judge scores against.
What's not there is the excluded document versions, a check that the answer stayed within the allowed sources, and abstention scored separately from omission. The repo is open source, you would be very welcome to add these. A pull request with a few wrong-version questions and a separate abstention score would make it much better learning resource for everyone.

Thanks and glad this helps. First of all it's 43 questions rather than 43 pages, so a lighter 2am read than it sounds. On faithfulness, the judge splits the answer into individual claims and checks each one against the 5 chunks the model was given, then returns the counts plus unsupported claims. The key rule in the prompt is that a claim that's true in general but not stated in the chunks counts as unsupported, so the model gets no credit for outside knowledge. Citations aren't counted as claims.
I don't have a separate contradiction flag. A contradicted claim is just counted as unsupported, so it lowers faithfulness the same way an invented one does. To check the judge wasn't too lenient, I gave it a deliberately wrong answer with two invented claims and it caught both faithfulness (0.33) and hallucination (1).

Thanks one note on [precision@5](mailto:precision@5). It divides the relevant chunks in the top 5 by 5, and most questions have only 1 to 3 relevant pages. A question with one relevant page can score at most 1/5 = 0.20 even with a perfect ranking. Across the 41 answerable questions, the best possible average is about 0.38, so, 0.34 is roughly 90% of the maximum.
On the dataset, an LLM drafted the questions from the PDF's and I checked and edited each one since 43 questions are not a large number. You are right that strictness is tricky. My first checked list had too many small details and marked good answers down, so it cut them to important facts. And yes, I checked every Page number with [p](mailto:precision@5)df[p](mailto:precision@5)lumber so it matches what the vector store returns.

Thanks, for your comment.

On trusting the 0.97, four answers weren't fully correct, and two show different failure modes. In Q40(pci dss vs fca authentication) only 1 of 3 relevant pages reached top 5, so the answer missed pci dss's two-factor rule for remote access, a retrieval failure.

In Q22 (methods for safeguarding funds) all the right pages reached the top 5, but the answer still left out, a generation failure. Both scored 1.00 on faithfulness, which is why I'd never rely on faithfulness alone. I also tested the judge with a deliberately wrong answer containing two invented claims, and caught it both.
The dataset, the code, and all saved results are in the repo: https://github.com/saurabhkamal/RAG-ReRanker-Comparison . Anyone can regenerate the report from the saved results. Re-running the full pipeline needs the vector collection from my ingestion plus Cohere and Jina keys.

Yes, I can, but why not clone the repo, experiment with it, add a few things from your side, and compare it? In this way, you will understand more.

Comments are written so that people can follow and understand them easily.

Well, I don't rule out that LLM assistance was used, but it doesn't matter as long as one can explain what is built and how it is built. Even in companies, they want to understand you, your approach, your thought process, your problem-solving abilities.

By the way, have you checked the video? It's all about evals: Hit Rate@5, Precision@5, Recall@5, MRR, MAP, NDCG@5, Faithfulness, Hallucination rate, Answer relevance, Context relevance, Answer correctness.

And if you can understand dense topics such as these without comments, then hats off too you man, and do explain others also. From my side, comments and videos are created to understand the topic.

I hope that answers.

Thanks for the suggestion, and my data partly backs it up. In the 43-question evaluation, 24 questions need more than one page. With Cohere, the full set of relevant pages reached the Top 5 in 18 of them. Where it didn't, correctness usually held, because the missing page was only partly relevant, but two answers lost key facts. The clearest was a cross-document question about authentication in PCI DSS and the FCA, where only 1 of 3 pages made the Top 5 and the answer missed one of three key facts. Hit@5 counted it as a success. I don't have a deliberate rule plus an exception question or an all required evidence present metric yet. Both are good additions, especially with near-duplicates left in the pool to compete for the 5 slots. That's the test that would show whether a reranker improves coverage or just promotes one good passage.

Thanks, really appreciate the star. I haven't tested bge-reranker-v2-m3 or a local Jina model yet. The Jina results are from jina-reranker-v3 through the API, and MiniLM was the only local model. You are right that it shows its age, and a modern local reranker would make the comparison much fairer, so that's on my list. On my laptop's CPU the larger BGE models would be slower than MiniLM, so a GPU would make that test more useful.

On latency, reranking the 20 chunks averaged 0.81s per question for Jina and 1.04s for Cohere, network call included, versus 1.29s for MiniLM running locally on CPU and about 4.5s for the LLM reranker. So in my setup the APIs weren't the slow option; they were faster than local MiniLM and placed the answer in the Top 5 on all 10 questions. These are averages over just 10 questions, so worth measuring on your own pipeline before showing stakeholders.

On latency, the surprise was that local wasn't faster. Reranking 20 chunks took about 1.29s per question for MiniLM running on my laptop's CPU, versus 1.04s for Cohere. On a GPU MiniLM would be much quicker, but on CPU the API was faster and far more accurate. The LLM reranker took about 4.5s

Thanks, good thought on truncation, so I checked it. The answer chunks for the two questions where MiniLM failed were 270–417 tokens including the question, all under the 512 limit, so the model read them in full. My chunks are capped at around 1,500 characters, so most fit comfortably, and the answer chunks did. That means truncation wasn't the cause. Fair point on Hit@5. Cohere and Jina both scored 10/10, so it can't separate them. My evaluation framework already computes MRR and NDCG@5 over 43 questions, but so far only for Cohere. So, running Jina could be the obvious next step.

Good point. I haven’t separated them yet. The scrambled-text explanation comes from one case that I traced in detail (PCI DSS Requirement 3, where two columns and a sidebar were merged by extraction and the MiniLM cross-encoder dropped that chunk from 1st to 8th), so it’s an observation rather than measure effect. I agree splitting chunks into clean vs scrambled would show how much is reranking and how much is extraction, and would be a good step.

On questions, the reranker comparison used 10 questions acoss 5 pdfs, all four rerankers plus plain vector search. The full evaluation framework uses 43 questions, direct, multi-page, vague, cross-document and unanswerable, with 6 retrieval and 5 generation metrics, but so far only with Cohere reranking.

Thanks that matches what I saw. The small cross-encoder in my comparison (MiniLM) behaved the same way, it favoured overview pages that repeated the question’s words and pushed the answer pages out of the Top 5 on two questions, doing worse than no reranking at all. Cohere, Jina, and the LLM reranker judged real relevance.

Good to know the 4B Qwen3 reranker crosses that line and would be a nice local option to add to the comparison.  Have you compared the 4B against the 8B, or measured latency on your GPU?

Hi, Why does it look like spamming? Shared here once. Sharing, what I build so others can try...and cross-post feature is a completely legitimate.

Interesting

Dude, you are taking it personally. It is a tutorial and with every new one, it would keep on improving, but still every example you have listed is a weakness of BM25 used on its own. Nobody here is proposing that. This tutorial combines BM25 with vector search and an LLM reranker because each one covers the other's gaps, and then it measures the result instead of assuming it. Meaning-based questions are handled by vector search, and word order can be handled by the reranker, which reads every candidate in full. Page 1 vs page 99 doesn't apply, because the search runs on single-page chunks, not whole documents.

Good night

This isn't an agent pipeline. It tests retrieval on questions the way real users actually type them so full natural questions are the right test here, not keyword lists. The extra words also matter less than it might seem...BM25 gives almost no weight to common words like which, does, or because they appear everywhere....the real work is done by stripe, SaaS, and subscription.. BM25 already treats those terms as "and/or. Embedding models are trained on natural sentences, so full questions are what they handle best. And since all four methods got exactly the same questions, the wording the didn't favour any of them. The comparison between them stays fair. I do agree that five fairly different documents is a small collection. A large set of similar documents would be a different test.

I used 1,500 characters per chunk with a 200-character overlap. Chunks are cut one PDF page at a time, so a chunk never spans two pages. That's also why the answers can be checked by page, and the exact same chunks go into both Qdrant and BM25. RRF uses k = 60.

You're right that one big results table can hide where each method does well. In my project, BM25 did well on the financial documents, probably because many questions use exact terms like "PCI DSS Requirement 3". Vector search should do better when a question uses different words from the document. But with only 10 questions and no labels, I can't prove that yet.

Thanks again, will make the comparison much more useful.

Thanks for your comment, but have you had a chance to clone the repo or watch the video? A few of your highlights are already covered:

- eval/run_comparison.py already prints hit/miss and rank for every question, for all four variants. They have listed one variant after another rather than side by side, but you can already see which questions the reranker wins on, and checking whether those wins line up with exact-term questions (a fee name, a requirement number) is a good idea, and I will look into it.

- Truncated chunks doesn't happen here because chunks are capped at 1500 characters when the PDFs are ingested, and the reranker gets the full chunk text with nothing cut off.

- For cost and latency, I agree, and the README calls it out. With 10 candidates per question, the reranker makes 10 LLM calls per query, so a full run takes minutes, not seconds. On the keyword-heavy documents plain RRF may well be enough, and that's exactly the trade-off this comparison is meant to show.

- I agree about the 10 questions, but see the results as showing a direction, not a final proof. In the third video, I have added 30 to 40 questions, but focus will be on on hitrate, precise, recall, MRR, MAP, NDCG, and generation metrics such as faithfulness, hallucination, answer_relevance, context_relevance, and answer correctness.

Thanks

Thanks, you are right about the maths. With 10 questions, every question is worth 10 points of hit rate, so small gaps don't prove much.

On MRR, Hybrid (RRF) went from 0.583 without the reranker to 0.80 with reranker. So, I think reranker is genuinely helping, but the hit-rate gap (0.800-1.00) comes down to just two questions, so I am treating the results as a strong signal, not proof.

This is the first part of the video. In the third part I have added 30 to 40 questions but mostly they will be focusing on hitrate, precise, recall, MRR, MAP, NDCG, and generation metrics such as faithfulness, hallucination, answer_relevance, context_relevance, and answer correctness.
I will see what I can do more, but thanks for the push.

I would say code is on GitHub, fork it or clone it and try to do it from your end, and lets discuss then.

Thanks for your thoughtful comment. Here's, how I ran the comparison.

I tested all four approaches, Vector only, BM25 only, Hybrid with RRF, and Hybrid + Reranking, in exactly the same way; the same 1507 chunks from five financial and regulatory PDFs, and the same 10 questions. For each question, I found the correct answer myself by locating the exact PDF and page. Every approach was then scored with Hit Rate, MRR, and NDCG. Nothing was pre-filtered by any search method, so the test doesn't lean towards BM25.

Honestly, I expected BM25 to struggle too. But on these documents it was the best single method, with a 0.900 hit rate against 0.700–0.800 for vector search. I think it's because regulatory documents use a lot of exact terms, like requirement numbers, regulation names and acronyms such as PCI DSS, and keyword search handles those well.

In my pipeline, each search method only sends its top 10 results into fusion. A document ranked 500th or 80th never reaches RRF, so that case doesn't come up the way you describe. Fusion also isn't the final step: an LLM reranker then reads each candidate against the question and scores it, which is where keyword-stuffed content should get pushed down. With the reranker added, the system found a correct answer for all 10 questions.

I agree with you:

- RRF does discard the original scores. That's a known trade-off and the reason I added a reranker.

- 10 questions is a small test, so I see the results as a strong signal, not final proof.

- Normalizing scores, like your query-energy approach, is a valid alternative to rank-based fusion.

I would genuinely like to see how CoreQuarry performs on the same questions. The code and test set are public, so you are welcome to try it and compare.

Thanks

Thanks for your thoughtful comment. Here's, how I ran the comparison.

I tested all four approaches, Vector only, BM25 only, Hybrid with RRF, and Hybrid + Reranking, in exactly the same way; the same 1507 chunks from five financial and regulatory PDFs, and the same 10 questions. For each question, I found the correct answer myself by locating the exact PDF and page. Every approach was then scored with Hit Rate, MRR, and NDCG. Nothing was pre-filtered by any search method, so the test doesn't lean towards BM25.

Honestly, I expected BM25 to struggle too. But on these documents it was the best single method, with a 0.900 hit rate against 0.700–0.800 for vector search. I think it's because regulatory documents use a lot of exact terms, like requirement numbers, regulation names and acronyms such as PCI DSS, and keyword search handles those well.

In my pipeline, each search method only sends its top 10 results into fusion. A document ranked 500th or 80th never reaches RRF, so that case doesn't come up the way you describe. Fusion also isn't the final step: an LLM reranker then reads each candidate against the question and scores it, which is where keyword-stuffed content should get pushed down. With the reranker added, the system found a correct answer for all 10 questions.

I agree with you:

- RRF does discard the original scores. That's a known trade-off and the reason I added a reranker.

- 10 questions is a small test, so I see the results as a strong signal, not final proof.

- Normalizing scores, like your query-energy approach, is a valid alternative to rank-based fusion.

I would genuinely like to see how CoreQuarry performs on the same questions. The code and test set are public, so you are welcome to try it and compare.

Thanks

Fair point, thanks for pushing on it.
For embeddings, I have used gemini-embedding-2-preview, 3072 dims. 1500-char chunks, 200 overlap, top-5 by cosine in Qdrant, and gpt-5.6-sol does grading, rewriting, generation and validation.

This focuses on the architecture - crag loop, self-validation, and bounded retries. It's validated on a domain set of regulatory PDF's, not yet on a public benchmark.

The retry limits are parameters, and setting both to 0 gives a plain-RAG baseline on the same pipeline. A with/without comparison on a HotpotQA or FRAMES could be the right next step.
and the retry loops are bounded by parameters, so they can be switched off to compare against a no-retry baseline

Thanks! that was the whole point. A compliance tool that confidently guesses is worse than useless here., so i wanted it to actually grade its own retrieval and admit when the context doesn't support an support an answer.

On your question though, this project doesn't actually have a dedicated reranker. What is it has a relevance grading step - one batched LLM call that labels all retrieved chunks as relevant/irrelevant at once.

If you want an actual reranker benchmarked properly, I built a separate project for exactly that
dense + BM25 + Reciprocal Rank Fusion, then an LLM reranker on top, tested against vector-only/BM25-only/hybrid baseslines with Hit Rate, MRR, and NDCG:
https://github.com/saurabhkamal/Hybrid-Search-RRF-Reranking-System

https://www.youtube.com/watch?v=W_TmnscEupQ&t=40s

Here the reranker scores each candidate chunk individually - one model call per chunk, not batched, so it is noticeably slower than a single grading call by design. Quality-wise it is worth it.

Please go through the links both github and youtube video.

Thanks for your comment. Well, you are absolutely correct, but this is an MVP. The idea is to clone it or fork it and play around with it. See, everything can't be given on a platter. I can't release the client's complete project here and never will. Moreover, this is one of the projects where you can add value for the client.
You have to understand there are many types of RAG, and this is one of them; day by day, I would be releasing other TYPES OF RAG as well, with evals and reranking.

Everything in a startup is an idea and then implementation. The interesting phase that you are talking about comes afterwards, and the interesting part that you are talking about is already shipped to the client.

The core part here is to understand this and where you can implement it by making it seem sufficient by adding value.

Thanks