`server`: cancel prompt processing & non-streamed requests when connection closed#9679

Closed

ochafik wants to merge 19 commits intoggml-org:masterfrom

ochafik:server-cancel

Commits on Sep 29, 2024

`server`: support cancelling non-streamed requests
ochafik
committed
`tests`: allow artificial slowdown of sampling for tests
ochafik
committed
`server`: test request cancellation (WIP)
ochafik
committed
`server`: rm superfluous is_alive check in streamed code
ochafik
committed
`server`: simplify handle_tasks signature
ochafik
committed
`server`: fix cancel tests
ochafik
committed
`server`: fix seed in tests (comma creates a tuple)
ochafik
committed
Update cancel.feature
ochafik
committed
`server`: update log
ochafik
committed
`server`: speed up cancel test setup
ochafik
committed

Commits on Oct 4, 2024

server: fix error status
ochafik
committed
`server`: introduce supposedly lighterweight is_alive in httplib (https://github.com/yhirose/cpp-httplib/pull/1956)
ochafik
committed
`server`: support cancellation of prompt processing
ochafik
committed
`server`: mime nit
ochafik
committed
`server`: avoid calling sink.is_alive() after it died 🧟‍♂️
ochafik
committed
Delete tests/.DS_Store
ochafik
committed
`server`: use (new) Request::is_alive as set_content_provider called after status / headers sent
ochafik
committed
`server`: fix disconnection logic in test (before post response headers)
ochafik
committed

Commits on Oct 5, 2024

reinstate trailing \n
ochafik
committed