queue in the api calls?

[batrlatom]

Hello,

I am interested in emulating the standard behavior of the Gradio app, which stores queries in a queue and processes them in sequence. Currently, the app can only handle one query at a time, and any additional tasks added while a task is already in progress will be lost and not processed.

As I plan to use the web UI as a backend for an online service and expect to receive queries from multiple users, I am wondering if there is a way to process queries from more people concurrently. Is this feasible?

[westmancurtis]

I was looking for this ages ago but instead I just implemented a queue in the code I use to call Web UI. Might offer you more flexibility and control in the long run.

[westmancurtis]

Yeah, I have a client app that speaks to a server app that calls webui directly. It's likely more complicated than it needs to be but it allows me to traffic requests directly back to the users fairly easily.

[speaknowpotato]

Yeah, I have a client app that speaks to a server app that calls webui directly. It's likely more complicated than it needs to be but it allows me to traffic requests directly back to the users fairly easily.

what does the client app look like? is it still the original gradio or your own UI? thanks!

[souravsk]

I have also created a new frontend and I'm using WebUI API to generate images but the problem is that when multiple users make requests to generate images it all goes in a queue I want them to parallelly process at a time.
@batrlatom @racinmat @vladmandic @westmancurtis @w-e-w

[w-e-w]

parallelly is not supported

or at least if it's going to support it lots of things needs to be changed

one reason is that within a single "job" the loded the current load a checkpoint can change
such as when encountering refine hires fix checkpoint model override request
just think what will happen when request_1 is in the middle of generation
and a second request comes in asking to change the model

also extensions can monkey patch webui on the fiy and unpatch at the end of the job
if you have to parallelly request there's possibility of different extensions patching different parts of where webui
or extension patching the same thing multiple times and causing a breaking stuff

if you want it parallel request, launch a second instance of web UI
if the reason you want to do so is for performance
then I recall someone testing running multiple instances in parallel the performance benefit is negligible

I believe there's also a solution that allowing you to chain multiple instances of web you are together
https://github.com/papuSpartan/stable-diffusion-webui-distributed

[souravsk]

If I want to provide text to images as services then for every user I have to have one instance with a load balancer on top of that but I don't see it to be cost-effective.

OR you an any there solution for that to handle multiple users?

[vladmandic]

there is cmdline flag --gradio-queue, but usage is experimental at best.

[batrlatom]

there is cmdline flag --gradio-queue, but usage is experimental at best.

i see, i will try it. thanks!

[batrlatom]

ok, so it looks like it works. I have one more question though. I am checking the status of generated image and rendering intermediate steps. But now, all connected clients see actually generated image. Is it possible to show the progress only for user whom image is actually generated?

[vladmandic]

there is no user management in gradio at all, only initial authentication to allow/disallow the http call.
if i'm building a multi-user webui server, i'd have a api layer in-between anyhow, i wouldn't want to expose gradio api to end-users.

[batrlatom]

ok, i will add some middle layer to my app. thanks!

[racinmat]

The flag does not do anything now https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/cmd_args.py#L103