u/Affectionate_Slip654

Great suggestion, here's what's in place so far, two unanswerable questions (stripe's per-transaction fee for iDEAL, and the maximum pci dss non-compliance fine) which the system correctly declined to answer. Each question also stores its relevant source pdfs and pages, and a checklist of key facts the judge scores against.
What's not there is the excluded document versions, a check that the answer stayed within the allowed sources, and abstention scored separately from omission. The repo is open source, you would be very welcome to add these. A pull request with a few wrong-version questions and a separate abstention score would make it much better learning resource for everyone.

Thanks and glad this helps. First of all it's 43 questions rather than 43 pages, so a lighter 2am read than it sounds. On faithfulness, the judge splits the answer into individual claims and checks each one against the 5 chunks the model was given, then returns the counts plus unsupported claims. The key rule in the prompt is that a claim that's true in general but not stated in the chunks counts as unsupported, so the model gets no credit for outside knowledge. Citations aren't counted as claims.
I don't have a separate contradiction flag. A contradicted claim is just counted as unsupported, so it lowers faithfulness the same way an invented one does. To check the judge wasn't too lenient, I gave it a deliberately wrong answer with two invented claims and it caught both faithfulness (0.33) and hallucination (1).

Thanks one note on [precision@5](mailto:precision@5). It divides the relevant chunks in the top 5 by 5, and most questions have only 1 to 3 relevant pages. A question with one relevant page can score at most 1/5 = 0.20 even with a perfect ranking. Across the 41 answerable questions, the best possible average is about 0.38, so, 0.34 is roughly 90% of the maximum.
On the dataset, an LLM drafted the questions from the PDF's and I checked and edited each one since 43 questions are not a large number. You are right that strictness is tricky. My first checked list had too many small details and marked good answers down, so it cut them to important facts. And yes, I checked every Page number with [p](mailto:precision@5)df[p](mailto:precision@5)lumber so it matches what the vector store returns.

Thanks, for your comment.

On trusting the 0.97, four answers weren't fully correct, and two show different failure modes. In Q40(pci dss vs fca authentication) only 1 of 3 relevant pages reached top 5, so the answer missed pci dss's two-factor rule for remote access, a retrieval failure.

In Q22 (methods for safeguarding funds) all the right pages reached the top 5, but the answer still left out, a generation failure. Both scored 1.00 on faithfulness, which is why I'd never rely on faithfulness alone. I also tested the judge with a deliberately wrong answer containing two invented claims, and caught it both.
The dataset, the code, and all saved results are in the repo: https://github.com/saurabhkamal/RAG-ReRanker-Comparison . Anyone can regenerate the report from the saved results. Re-running the full pipeline needs the vector collection from my ingestion plus Cohere and Jina keys.

Yes, I can, but why not clone the repo, experiment with it, add a few things from your side, and compare it? In this way, you will understand more.

Comments are written so that people can follow and understand them easily.

Well, I don't rule out that LLM assistance was used, but it doesn't matter as long as one can explain what is built and how it is built. Even in companies, they want to understand you, your approach, your thought process, your problem-solving abilities.

By the way, have you checked the video? It's all about evals: Hit Rate@5, Precision@5, Recall@5, MRR, MAP, NDCG@5, Faithfulness, Hallucination rate, Answer relevance, Context relevance, Answer correctness.

And if you can understand dense topics such as these without comments, then hats off too you man, and do explain others also. From my side, comments and videos are created to understand the topic.

I hope that answers.

Thanks for the suggestion, and my data partly backs it up. In the 43-question evaluation, 24 questions need more than one page. With Cohere, the full set of relevant pages reached the Top 5 in 18 of them. Where it didn't, correctness usually held, because the missing page was only partly relevant, but two answers lost key facts. The clearest was a cross-document question about authentication in PCI DSS and the FCA, where only 1 of 3 pages made the Top 5 and the answer missed one of three key facts. Hit@5 counted it as a success. I don't have a deliberate rule plus an exception question or an all required evidence present metric yet. Both are good additions, especially with near-duplicates left in the pool to compete for the 5 slots. That's the test that would show whether a reranker improves coverage or just promotes one good passage.