u/capestart_tech

Yeah, the number of chunks is worth checking too. If the model answers correctly with one passage but struggles when the others are added back, that's a useful clue. Testing different top-k values could help narrow down whether the extra context is causing the problem.

Good point about checking what actually reaches the model. A passage showing up in the retrieval results doesn't necessarily mean it made it into the final context. Comparing the two would help catch cases where reranking or filtering drops the right information before generation even starts.

For interruptions, I'd look at how fast you can cancel the current TTS output and stop the agent from continuing with a response the user no longer wants. It is also useful to keep track of what was actually said, as the user may cut off mid-sentence.

I’d test that early with actual conversations in the target language. The timing can feel very different the moment people start interrupting naturally.

If an agent compresses its context while waiting for someone to approve an action, that approval must be linked to the original request. Otherwise you might be voting on something without the full context of what the agent was going to do.

I'd rather store this approval status outside of the model context with enough information to safely resume or deny the action.

We shared this without a specific contract dataset in mind, so we don’t have an amendment count to provide.

Your point about merging the amendments into the current version is an interesting one though. Especially cases where an amendment changes a definition that touches several clauses. It makes sense to flag these for manual review, as missing one could impact several answers.

The snapshot approach is interesting especially for rollback. I get it, it would make debugging easier if retrieval behavior changed after an update. As you build up those snapshots do you encounter a problem of storage or indexing costs?

It seems like a sensible safeguard to have the old version available until the replacement passes checks. Do you swap the active version all at once after indexing is done, or is there some validation step before it becomes searchable?

The stable ID approach is a good idea, especially when files are renamed or moved between folders. What do you do when someone uploads a new version of a document as a new file instead of replacing the old one?

If you are already familiar with langchain, I would probably dig deeper into langgraph before trying to learn every framework on that list.

If anything, for production work I'd spend as much time learning how to handle state, retries, tool failures, tracing and evals. Frameworks will continue to proliferate, but those problems will be there no matter what framework you choose.

Mixed for us as well. Still, RAG makes sense when the job is mostly retrieval and the workflow is well defined.

Agents are more useful when there are multiple steps involved, especially when the system needs to make a decision on what to retrieve or what tool to call next. But that also gives you more to debug and evaluate.

We'd only add an agentic layer if the workflow really needed one.