u/hlcAlex

You caught my architecture's exact trade-off!

Truth is: my regex layer does 80% of the heavy lifting (zero cost, instant), but it's heavily optimized for EN/CN. For a Korean sentence, the regex rules will likely do nothing and just pass it through.

This means for Korean, the system will rely almost entirely on the LLM/NLI layer as the safety net. Since the LLM reads the full context, it should still catch the missing condition across languages.

The regex layer saves costs on high-volume EN/CN docs. The LLM layer is my universal fallback. If you try a Korean test case and the LLM misses it, I definitely need to improve my prompts for CJK languages. Give it a try!

Exactly.Tried the same hardcoded-marker preprocessing first-worked fine on early docs,but kept breaking as soon as I hit constructs that have zero structural signal at all.Gave up patching the splitter

One thing that bugged me though:nobody seems to have a second layer after the rule layer.Block looks syntactically complete,only reads as dependent when yu see it next to the prior chunk--rule layer can never touch this,but LLM with full_doc context catches most of it.Verifier I built is two-stage by design for exactly this reason

Hey! Here is my side project:https://rag-verifier.streamlit.app/

it's an open-source verification layer for RAG(AI document QA).it catches 'fake citation'-where the AI quotes a real sentence,but it doesn't actually support the claim

Don't judge the uI,it's just a quick Streamlit wrapper

How to test in seconds:Just click the preset button 'Test Case:Missing Context', then hit Verify,it will show yu how a real quote can be misleading if it loses its condition

As a develope,i'd love to know:Does the error verdict/reason make sense to you?

That's why I don't trust the parser blindly.if bad reading order creates a scrambled chunk(like missing context or starting with Therefore...),my chunk validation layer flags it

Built a quick live demo to test this exact 'silent failure' scenario. you can paste a broken chunk and see if it gets caught:https://rag-verifier.streamlit.app/

This is incredibly helpful,I actually just went through my codebase against your points,Here's where I stand:What I'm alredy doing:Out-of-context quotes:I feed the full source_text to the judge,not just the snippet,and I have a hardcoded rule,if the quote is the result of an 'if A,the B'condition,it fails unless condition A is included in the citation

No blanket passes:I use a 3-state verddict to trigger different recovery paths downstream,so a sigle citation never valiates the whole answer

Where you hit real gaps:Atomic claims:I currently rely on the caller to pass pre-split atomic claims,I don't do end-to-ent sentence splitting in this layer,so you're right-I shouldn't market it as full answer-level evaluation yet

Two-passage claims:My API currently only accepts a single string for citation,if a claim needs two paragraphs to support it,the exact-match layer will fail,This is top of my fix list

Adversarial set:I only have 5 failure types right now

Your best catch:Separating false accepts from false flags.I have a confusion matrix but don't surface these numbers separately

I built a live demo based on this rule-first logic(link below),Would love your thoughts on the verdict logic!

https://rag-verifier.streamlit.app/

A text-match check will 100% miss a deprecated or wrong-region quote.Version,date,and scope must be hard metadtaa filters,scored separately from text entailment,

My current demo only checks text-to-text,but this is top of my roadmap,just built it based on this thread,would love your thoughts:https://rag-verifier.streamlit.app/

your point about missing conditions being a chunking problem at ingestion is spot on,I just built a quick live demo based exactly on this rule-first+degradation (if no LLM key is provided,it degrades and explicitly marks the semantic check as skipped,not tested,rather than blindly passing it)logic,would love your thoughts if you have a minute:https://github.com/hlc0000/rag-verifier

In my dem if you just leave the citation field empty,the NLI check will run the claim directly against the source text,Feel free to test it here:https://rag-verifier.streamlit.app/

I'd say messy starts when structure/reading order isn't reliably recoverable before chunking,not when it's "not clean text",two-column pDFs and tables qualify,HTML/Markdown/code are structured,so not automatically messy,ocR is messy even earlier,So it's a parsing-layer question,not a ccontent-type-type question

charts it can't extract anything, just classifies them as picture and counts them,nothing goes into the text,tables are fine though,tested those thoroughly

tested it a lot,Two-column papers and long docs are fine,it's decent but not reliable blindly,i run apymupdf reconciliation check on the output

Haha,I just started using Docling as part of my doc parsing

Depends on the content,plain text,PDFs,code?overlap is fine for simple text,but for messy docs I'd start structure