Best practices when preparing documentation for ingestion (for Q&A over documents)

[dukemagus]

Hi

I intend to put two complementary bodies of knowledge in a langchain app: one is a wiki (mostly static) and the other is a blog-like news feed with updates to the content of the wiki

• Which are the best practices to preprocess the documents and minimize hallucination or wrong answers (and, if possible, reduce API calls)?
• Are there any recommended preprocessor applications, prompts or models that prepare the page, strip useless data and uses the metadata of the page and strucute of documents to help in the index (wiki page categories and the site map/tree, page indexes, etc).
• Does the document processing identifies redundancy (example: 10 pages talk about an engine, every page has a couple paragraphs that are nearly identical since they were made using the same template, but the individual specs in each page is different)? How does it deal with redundant terms and phrases?
• Are there any specific actions that need to be avoided when passing the pages through a script or manually adding the text to the body of knowledge?
• Are the databases generated by langchain static and need to be rebuilt every time an information is added or changed or you can load a new document that changes the answer of an old question and use it right away?
• Can you create and fine tune "weights" or is there another way of progressively improve the answers given by the chain. does it have a standard function to "downvote", "correct" an answer or inform that the right answer is in a different page?

Thanks for the help. I tried a very basic example with a couple of manuals before and the AI stumbled repeating the definition of an item rather than what should be its unique data. :(

[UncIeRick]

Hi dukemagus,

I wonder if you have any insights about this question now? Thank you!