Skip to content

big indexing - openai requests may break #4

@mvpmaster

Description

@mvpmaster

when i tried to do index many big documents on default mode (openai),

at some point completions request fail on 404 nginx error and fail whole script, and it was not able to save current state

File "/Desktop/projects/PathRAG/PathRAG/operate.py", line 317, in _process_single_content
glean_result = await use_llm_func(continue_prompt, history_messages=history)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Desktop/projects/PathRAG/PathRAG/utils.py", line 119, in wait_func
result = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "Desktop/projects/PathRAG/PathRAG/llm.py", line 483, in gpt_4o_mini_complete
return await openai_complete_if_cache(

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 404 Not Found"

so i started from beginning and making backup of index after each document processing
(but then wrote some fast fix code)

gpt_4o_mini_complete - need check , if something go wrong

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions