Possible to get server status (busy, idle etc.) and other info from api?

[mercurial-moon]

Is it possible from the kobold cpp api

1. To query the server and see if its busy doing a generation or just idle (no current requests). Just a yes/no response or something on these lines.
2. In case I send a text generation request to /api/v1/generate how do I know that the response is partial or complete. OpenAI has something like finish_reason does Kobold api have something similar?
3. Know the IP address of the user currently for whom the text generation is currently processing.

[LostRuins]

1. Yes, this is possible. You can get all performance and idle/busy information at the endpoint /api/extra/perf
2. Yes this is possible. KoboldCpp API also has a finish_reason field, and works in the same way
3. This is not currently available

[mercurial-moon]

Also is structured output generation supported only via OpenAI endpoints or also on the KoboldCpp endpoint?

[LostRuins]

KoboldCpp endpoint supports traditional tool calling but not structure outputs, as it's a completions based endpoint

[mercurial-moon]

Hi, i see the koboldcpp api has last_token_count is this the last text generation token count or the last prompt process token count? From my initial tests it seems to be amount of prompt tokens processed count.
Is there a way to get the
a. the amount of tokens that were sent to the llm in the last query (question).
b. the amount to tokens actually processed from the last query (due to prompt caching) - this i think you already provide via last_token_count (part of question that was actually used)
c. The amount of tokens that were generated by the last query (answer).

[LostRuins]

last_process_time = time to process prompt
last_eval_time = time to generate new tokens
last_input_count = number of tokens in input processed
last_token_count = number of tokens generated in output response.

Yes the names are slightly confusing, unfortunately they were set long ago so its hard to change

[mercurial-moon]

Thanks, I don't see last_input_count on this page https://lite.koboldai.net/koboldcpp_api in the KcppPerf section... I guess it might not be updated.
Also last_input_count is the number of input tokens processed? so if 10000 tokens came in and only 2000 were processed will last_input_count indicate 2000?
Is there a way to get the actual number of input tokens that came in with the query eg. 10000 in the above example.

[jabberjabberjabber]

You should be able to figure it all out from reading this code.

It was written a while ago but the basics should be covered enough to get you familiar with it.