How to make the response (inference) faster? I have built a chatbot based on GraphRag but it is too slow.

[alchem-ist]

Also, what is the easiest way to change the system prompt?

[MisterAndry]

How to change the system prompt - monkeypatch.
For example, I have defined a new local search pormpt here -graphrag_prompts.LOCAL_SEARCH_SYSTEM_PROMPT

import graphrag.query.structured_search.local_search.system_prompt
import graphrag_prompts

def local_search(text):
    # import should be after patching prompt
    from graphrag.query.cli import run_local_search
    return run_local_search(
        config_filepath=None,
        data_dir=None,
        root_dir="/Users/cheat/PythonProjects/graphrag_test/chat-bot",
        community_level=1,
        response_type='Single Sentence',
        streaming=False,
        query=text
    )


def patch_prompts():
    setattr(graphrag.query.structured_search.local_search.system_prompt,
            "LOCAL_SEARCH_SYSTEM_PROMPT", graphrag_prompts.LOCAL_SEARCH_SYSTEM_PROMPT)


patch_prompts()
answer, ctx = local_search("Your question here")

[adrianzartis]

I think streaming would help in your case, is there a reason why it set to false? Alternatively you could use a faster model with a fast inference API like Groq for example. In that case you would have to create a new class that inherits the GraphRAG LocalSearch and include only the contex_text in the SearchResult, then you can pass that context to your faster API and do streaming. You could also try improving your retriever by either adding metadata or even fine-tuning your own embedding model.