[Question] What makes a "good" node?

[nshern]

Hello! I have a general question I hope someone can help me with:

Say I want to do RAG on documents that are meeting notes that are structured so that it is a bunch of verbatim statements from people, structured like

person 1: bla bla bla, 11:46
person2: bla bla bla , 11:47
person1: blablablablabla, 11:50
...
person3: blabla, 14:45

i.e. a structure that is person, statement, timestamp

Is it a good idea to define each node so that it is a single statement, together with metadata being name of person and time and with no chunk overlap? Or should I just let the node parser do it automatically?

Where can I find some information regarding this and what makes up a "good node"?

[dosubot[bot]]

🤖

Hey there, @nshern! Nice to see you back with more questions. Hope you've been doing well!

In the context of RAG on documents structured as verbatim statements from people, it would be a good practice to consider each statement as a single node with metadata. This approach is supported by the SentenceWindowNodeParser class in the LlamaIndex repository, which splits a document into nodes, with each node being a sentence. Each node contains a window from the surrounding sentences in the metadata.

The SentenceWindowNodeParser class has several parameters that can be adjusted according to your needs:

• sentence_splitter: This is a function that splits text into sentences. You can provide your own function or use the default split_by_sentence_tokenizer.
• window_size: This is the number of sentences on each side of a sentence to capture. The default is 3.
• window_metadata_key: This is the metadata key to store the sentence window under. The default is "window".
• original_text_metadata_key: This is the metadata key to store the original sentence in. The default is "original_text".

The SentenceWindowNodeParser class also includes methods for parsing nodes and building window nodes from documents. The _parse_nodes method parses a document into nodes, and the build_window_nodes_from_documents method builds window nodes from documents.

In terms of what constitutes a "good node", it would typically be a node that accurately represents a single statement from a person, includes relevant metadata (such as surrounding sentences for context), and is easily processed by the rest of your application. However, the exact definition of a "good node" may vary depending on your specific use case and requirements.

You can find more information about the SentenceWindowNodeParser class in the LlamaIndex repository.

Sources

• llama-index-legacy/llama_index/legacy/node_parser/text/sentence_window.py
• llama-index-core/llama_index/core/node_parser/text/sentence_window.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.