Feature request: Possibility of interrupting requests natively in llama.cpp

[DanielusG]

Hello, I often find myself in the situation of testing models or making requests to models that take a long time to reason and answer, and perhaps I no longer need the answer. Since llama.cpp, unless you explicitly specify parallelism (which consumes much more VRAM), the requests are sequentials, so to send a subsequent message I have to:

• Force‑kill the server (losing any conversation cache and reloading the model in RAM-VRAM) or
• Wait for the model to finish responding (which may take a long time)

Therefore, in the llama‑swap fork I added the ability to interrupt running requests. The interrupt works by using an AbortController so that the llama‑cpp proxy notices the stop request and terminates the response as soon as possible.

Would it be useful to open a pull request to merge this feature, or is it outside the scope of this project?

Some images of actual implementation:

[mostlygeek]

Hasn’t stopping generation at the client been enough to cancel generation? If not, that’s a bug. I know llama-server will interrupt generation if the http request is closed on the client side. llama-swap will close its http request if the client closes theirs.

[DanielusG]

I know llama-server will interrupt generation if the http request is closed on the client side. llama-swap will close its http request if the client closes theirs.

You're right, but to make it work you need to implement an AbortController, it doesn't happen automatically just by closing the connection.
Also many apps do not natively support llama.cpp, so they do not implement a proper system to stop the generation and inference, To avoid being dependent on software implementation, this feature is more useful to me.
Since I am using open source software, I believe that if my modification can be useful to someone else, I am happy to share it.
So let me know if this could be a useful feature to implement in the main branch and I'll open a pull request :)

[mostlygeek]

You're right, but to make it work you need to implement an AbortController, it doesn't happen automatically just by closing the connection.

I'm saying it should happen automatically by closing the connection. Both llama-swap and llama-server has had cancelable requests since January.

ref:

• llama-server added it in server : implement cancellable request ggml-org/llama.cpp#11285.
• llama-swap has had it since [Bug?] When clients crash, llama-swap does not send a cancel request to the upstream llama-server #43 was closed

I think what you're offering is more of a force cancel option to abort current inflight/waiting requests. In order to do this llama-swap would have to maintain a list of inflight requests and cancel their contexts on demand. The data structure would also have to clean up completed requests as well.

[DanielusG]

First of all, sorry. Before opening this issue I didn't do extensive research to see whether the issue had already been addressed.

I went through some more detailed, targeted tests to identify when it occurs, and I noticed that, as I mentioned earlier, it seems to depend heavily on the client implementation. This time I think the abort controller is no longer relevant, because, as you correctly pointed out, the interruption happens automatically when the connection is closed.

Specifically, I tried:

• OpenWebUI – despite interrupt button, inference continues.
• N8n – as soon as I press the cancel button, inference stops correctly.
• RooCode – when I press the cancel button, inference does not stop.
• Cline – stops inference when the task is interrupted.
• As noted in the issue you highlighted, in a Python script that sends the request, pressing Ctrl+C also stops the inference.

The pattern is clear: the interruption of inference is not always guaranteed; it depends largely on how the client handles the request.

I primarily use OpenWebUI and RooCode, which is why I didn’t notice that in other contexts the interruption works correctly.

In any case, this feature (at least for me) is very useful. In my fork I’ve published the modifications; feel free to take a look at the implementation and see if it could be useful for this project.

Since I use llama-swap daily and this feature is handy, I’ll keep my fork updated with these useful changes, resolving any conflicts. Also note that all the tests pass.

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity.

[MikeLP]

Have the same issue. Unfortunately abort controller does not stop the inference despite closed connection. I'm using OpenAI async client. Ollama works fine btw.

[mostlygeek]

@DanielusG, @MikeLP - the reverse proxy code has been changed may handle edge case cancellation behaviour better now. Please test again if you can.

[MikeLP]

@DanielusG, @MikeLP - the reverse proxy code has been changed may handle edge case cancellation behaviour better now. Please test again if you can.

Looks like it is fixed. At least after some quick testing. I will keep you posted.

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity.

[bonswouar]

I still feel like the "force abort" feature from OP could be useful, as clients don't always allow to close a connection before generation is done and seeing how resource heavy those can be, this would be much cleaner than stopping the docker container (my current workaround).

@DanielusG wouldn't you want to propose your changes as a PR instead of maintaining a separate version?

[DanielusG]

I still feel like the "force abort" feature from OP could be useful, as clients don't always allow to close a connection before generation is done and seeing how resource heavy those can be, this would be much cleaner than stopping the docker container (my current workaround).

@DanielusG wouldn't you want to propose your changes as a PR instead of maintaining a separate version?

For me, it wouldn't be a problem. I simply avoided doing it before because I don't know if it fits with the development philosophy of this repository's maintainer. To avoid having a ghost PR, I haven't opened one yet. If it's okay with you, I can do it.