WIP: simple openai image generation api compatiple server

[leejet]

Example:

server

.\bin\Release\sd-server.exe -m v1-5-pruned-emaonly.safetensors -v

client

curl http://127.0.0.1:1234/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sd1.5",
    "prompt": "A lovely cat<sd_cpp_extra_args>{\"seed\": 357925}</sd_cpp_extra_args>",
    "n": 1,
    "size": "512x512",
    "response_format": "b64_json"
  }' \
  -o >(jq -r '.data[0].b64_json' | base64 --decode > output.png)

[Green-Sky]

What api spec is that? That is an odd way to inject extra parameters...

[leejet]

OpenAI’s documentation: https://platform.openai.com/docs/api-reference/images

Because many parameters are not supported in OpenAI’s standard API, in order to use sd.cpp within clients that support the OpenAI API, I chose to include the extra parameters directly in the prompt.

[wbruna]

"prompt": "A lovely cat<sd_cpp_extra_args>{"seed": 357925}</sd_cpp_extra_args>",

If the idea is to be able to type it into the prompt of an existing application, maybe it could be simplified a bit?

If the option names don't have colons, we can avoid the need for quotes (like HJSON):

"prompt": "A lovely cat<sd_cpp_extra_args>{seed: 357925}</sd_cpp_extra_args>",

A variant of the current prompt syntax would likely be able to reuse most of the current code. The LoRA syntax, for instance:

"prompt": "A lovely cat<sd_cpp_extra_arg:seed:357925>",

The Perchance website uses a variant of the (token:weight) syntax, disambiguated with extra colons:

"prompt": "A lovely cat(seed:::357925)",

[stduhpf]

If the option names don't have colons, we can avoid the need for quotes (like HJSON):

I think supporting standard json is important for ease of use, but if we could accept keys both with or without quotes, then why not.

[tarruda]

Because many parameters are not supported in OpenAI’s standard API, in order to use sd.cpp within clients that support the OpenAI API, I chose to include the extra parameters directly in the prompt.

Are there a lot of clients that depend on this OpenAI image generation API?

Even if this API style is widely used, I suggest to create your own API server that is better suited to expose sd.cpp functionality without worrying about OpenAI compatibility, and then add OpenAI compatibility as a layer on top of that, translating the OpenAI style API call to the native sd.cpp server API. This is similar to how llama.cpp recently added Anthropic Messages API compatibility to their llama-server.

If OpenAI compatibility is an absolute requirement, then why not add a new body parameter that cleanly encapsulates all sd.cpp parameters not supported by OpenAI API? Something like this:

curl http://127.0.0.1:1234/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sd1.5",
    "prompt": "A lovely cat",
    "n": 1,
    "size": "512x512",
    "response_format": "b64_json",
    "sd_cpp": {
         "seed": 357925
     }
  }' \
  -o >(jq -r '.data[0].b64_json' | base64 --decode > output.png)

Since "sd_cpp" property is optional, this will still be compatible with any existing OpenAI clients.

[stduhpf]

Even if this API style is widely used, I suggest to create your own API server that is better suited to expose sd.cpp functionality without worrying about OpenAI compatibility

@tarruda I made this other server that runs image generation asynchronously, with a more feature-complete custom API: #367. Maybe it would be possible to add compatibility layers for other popular APIs like openAI or stable-diffusion-webui (not my priority though)

[wbruna]

I think supporting standard json is important for ease of use, but if we could accept keys both with or without quotes, then why not.

Sure, one does not exclude the other (I believe that's what the HJSON project proposes). And ease of use by humans is mostly what I'm concerned about: like the curl example, a JSON string inside quotes can easily go into escaping hell (I ran into that when implementing the sdgendefaults support for Koboldcpp).

Arguments coming from code could arguably better be served by a new dedicated parameter, like @tarruda mentioned... unless there's another reason to accept a mostly-standard format embedded on the prompt (say, making the parameters able to be generated by an LLM on an existing setup).

[wbruna]

I'd also suggest moving the top level to /openai/v1, to make the target standard more explicit, and avoid conflicts with future alternative standards (e.g. emulating A1111/Forge, as Koboldcpp does). If needed, "promoting" it to the top could be done by an optional command-line parameter, or even an intermediate webserver.

[leejet]

This PR is only meant to provide a simple OpenAI-compatible HTTP server, so that clients supporting the OpenAI image generation API — such as OpenWebUI — can conveniently use sd.cpp. Therefore, I’ve tried to keep the API as consistent with OpenAI as possible. Of course, this can’t fulfill every need, and additional API formats or sd.cpp-specific APIs may be added in the future.