slopfront
Reddit, read-only. No JavaScript. No cookies. No tracking.
preferences

Built a free open-source RAG verifier--because neither chunkers nor LLM judges check if the evidence is actually complete

1 points · submitted 4d ago by u/hlcAlex · Discussion

I built a free,open-source RAG verifier covering two gaps I don't see anyone else watching:

1.whether your chunks are actually self-contained

2.whether AI citations genuinely support their ckaims-not just whether the text exists somewhere in the source

Example:a contract Q&A confidently answered 15% liquidated damages-but the chunk it cited was just the consequent clause(Party B shall pay 15%).The trigger condition(only if Party B is over 60 days late)was in the previous chunk,which the splitter had cut right through,Actual penalty for the user's case was 0% LLM judge said 'citation exists word for word' and called it supported. Both were wrong

Rule layer is regex-based and deterministic-it catches explicit structural markers: dangling conditionals(if x without then) ,conjunctive starters(Then,Also,Meanwhile),Zero false positives across 20 real Chinese + English documents(1388 chunks total)

LLM layer does two things.Citation faithfulness is a three-way judgment-it specifically checks if your cited fragment is the consequent half of a conditional whose trigger is missing.Chunk independence passes full doc + chunk and asks whether the chunk actually holds without prior context.Rule layer can't catch blocks that are syntactically complete but semanatically dependent hat's LLM-only territory,and it only works if you pass the full context

Fail-open when LLM is unavailable. it never fakes a pass

Demo(paset your own chunk,test it live):https://rag-verifier.streamlit.app/

Library:https://github.com/hlc0000/rag-verifier

Curious ow others are handing chunk completeness in production-is anyone actually checking this,or is everyone just acception that chunkers will break conditional clauses and relying on retrieval + LLM judges to mask it?

Built a free open-source RAG verifier--because neither chunkers nor LLM judges check if the evidence is actually complete

4 comments · 100% upvoted · other discussions

comments

Glittering_Rain_4234 1 points · 4d ago

This is the exact type of edge case that makes me not trust most RAG setups in production

I ran into a similar thing last year building a policy doc bot, the chunker kept splitting "notwithstanding" clauses right in half and the LLM would happily cite the second half like it was the full story. We ended up writing a preprocessing step that just refused to split on certain structural markers like "provided that" or "subject to" but it was janky as hell and we still missed stuff

The three-way judgment for citation faithfulness is clever, most people just do binary "is this text in the source" checks and call it a day. Gonna poke around the demo later tonight

hlcAlex [OP] 1 points · 4d ago

Exactly.Tried the same hardcoded-marker preprocessing first-worked fine on early docs,but kept breaking as soon as I hit constructs that have zero structural signal at all.Gave up patching the splitter

One thing that bugged me though:nobody seems to have a second layer after the rule layer.Block looks syntactically complete,only reads as dependent when yu see it next to the prior chunk--rule layer can never touch this,but LLM with full_doc context catches most of it.Verifier I built is two-stage by design for exactly this reason

magicdoorai 1 points · 3d ago

Your 60-day example suggests a useful paired test that separates chunk completeness from whether the answer follows. This is a proposed synthetic fixture, not a test I've run against your library.

Use one invented source: 'If delivery is more than 60 days late, Party B owes 15% of the contract value. Otherwise no late-delivery penalty is due.' Ask: 'Delivery was 60 days late. Is 15% due?'

Run three evidence variants against the SAME answer, 'Yes, 15% is due':

  • Only 'Party B owes 15% of the contract value.' The quoted words exist, but the evidence is incomplete; it must not establish the answer.
  • The full source above. The chunk is now self-contained, but the answer is contradicted: exactly 60 is not more than 60.
  • The full source, changing only the case facts to 61 days late. In this simplified fixture, the answer is supported.

Then keep the full source and remove the delivery-delay fact entirely. The source is complete, but applicability is unknown. That case should not become supported just because retrieval returned the missing conditional.

I'd score 'missing prerequisite', 'known false prerequisite' and 'known true prerequisite' separately. Otherwise expanding a chunk can look like a verification improvement while the final answer still applies the clause to the wrong case. These expected labels are for the invented wording, not legal advice about the original contract.

For your fail-open path, run the same cases with the LLM unavailable and inspect what the caller receives. Preserve an explicit not_checked/unavailable state rather than letting absence of a finding look like supported. That tests the library-to-caller contract without claiming your implementation currently gets it wrong.

AI-generated test proposal based on the example in your post.

eksde_69 0 points · 3d ago

chunk completeness is what breaks most RAG setups, and bad splits don't get fixed at retrieval. your example is a good illustration.

OXR from SoMark is a open-weight parser that outputs reading order and layout coordinates in one pass, thanks to a separate layout model. that structural info should help your chunker keep conditionals intact. they have a