|
| 1 | +# OpenAI API image edit endpoint {#ovms_docs_rest_api_image_edit} |
| 2 | + |
| 3 | +## API Reference |
| 4 | +OpenVINO Model Server includes now the `images/edits` endpoint using OpenAI API. |
| 5 | +It is used to execute [image2image](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/image_generation) and [inpainting](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/image_generation) tasks with OpenVINO GenAI pipelines. |
| 6 | +Please see the [OpenAI API Reference](https://platform.openai.com/docs/api-reference/images/createEdit) for more information on the API. |
| 7 | +The endpoint is exposed via a path: |
| 8 | + |
| 9 | +<b>http://server_name:port/v3/images/edits</b> |
| 10 | + |
| 11 | +Request body must be in `multipart/form-data` format. |
| 12 | + |
| 13 | +### Example request |
| 14 | +``` |
| 15 | +curl -X POST http://localhost:8000/v3/images/edits \ |
| 16 | + -F "model=OpenVINO/stable-diffusion-v1-5-fp16-ov" \ |
| 17 | + -F "image=@three_cats.png" \ |
| 18 | + -F "strength=0.7" \ |
| 19 | + -F 'prompt=Three zebras' \ |
| 20 | + | jq -r '.data[0].b64_json' | base64 --decode > edit.png |
| 21 | +``` |
| 22 | + |
| 23 | +### Generate 4 variations of the edit |
| 24 | +``` |
| 25 | +curl -X POST http://localhost:8000/v3/images/edits \ |
| 26 | + -F "model=OpenVINO/stable-diffusion-v1-5-fp16-ov" \ |
| 27 | + -F "image=@three_cats.png" \ |
| 28 | + -F "strength=0.7" \ |
| 29 | + -F "prompt=Three zebras" \ |
| 30 | + -F "n=4" \ |
| 31 | + | jq -r '.data[].b64_json' \ |
| 32 | + | awk '{print > ("img" NR ".b64") } END { for(i=1;i<=NR;i++) system("base64 --decode img" i ".b64 > edit" i ".png && rm img" i ".b64") }' |
| 33 | +``` |
| 34 | + |
| 35 | + |
| 36 | + |
| 37 | +### Example response |
| 38 | + |
| 39 | +```json |
| 40 | +{ |
| 41 | + "data": [ |
| 42 | + { |
| 43 | + "b64_json": "..." |
| 44 | + } |
| 45 | + ] |
| 46 | +} |
| 47 | +``` |
| 48 | + |
| 49 | + |
| 50 | +## Request |
| 51 | + |
| 52 | +| Param | OpenVINO Model Server | OpenAI /images/edits API | Type | Description | |
| 53 | +|-----|----------|----------|---------|-----| |
| 54 | +| model | ✅ | ✅ | string (required) | Name of the model to use. Name assigned to a MediaPipe graph configured to schedule generation using desired embedding model. **Note**: This can also be omitted to fall back to URI based routing. Read more on routing topic **TODO** | |
| 55 | +| image | ⚠️ | ✅ | string or array of strings (required) | The image to edit. Must be a single image (⚠️**Note**: Array of strings is not supported for now.) | |
| 56 | +| mask | ❌ | ✅ | string | Triggers inpainting pipeline type. An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where `image` should be edited. Not supported for now. | |
| 57 | +| prompt | ✅ | ✅ | string (required) | A text description of the desired image(s). | |
| 58 | +| size | ✅ | ✅ | string or null (default: auto) | The size of the generated images. Must be in WxH format, example: `1024x768`. Default model W/H will be used when using `auto`. | |
| 59 | +| n | ✅ | ✅ | integer or null (default: `1`) | A number of images to generate. If you want to generate multiple images for the same combination of generation parameters and text prompts, you can use this parameter for better performance as internally computations will be performed with batch for Unet / Transformer models and text embeddings tensors will also be computed only once. | |
| 60 | +| input_fidelity | ⚠️ | ✅ | string or null | Control how much effort the model will exert to match the style and features, especially facial features, of input images. (⚠️**Note**: Not supported for now - use `strength` parameter. | |
| 61 | +| background | ❌ | ✅ | string or null (default: auto) | Allows to set transparency for the background of the generated image(s). Not supported for now. | |
| 62 | +| stream | ❌ | ✅ | boolean or null (default: false) | Generate the image in streaming mode. Not supported for now. | |
| 63 | +| style | ❌ | ✅ | string or null (default: vivid) | The style of the generated images. Recognized OpenAI settings, but not supported: vivid, natural. | |
| 64 | +| moderation | ❌ | ✅ | string (default: auto) | Control the content-moderation level for images generated by endpoint. Either `low` or `auto`. Not supported for now. | |
| 65 | +| output_compression | ❌ | ✅ | integer or null (default: `100`) | The compression level (0-100%) for the generated images. Not supported for now. | |
| 66 | +| output_format | ❌ | ✅ | string or null | The format in which the generated images are returned. Not supported for now. | |
| 67 | +| partial_images | ❌ | ✅ | integer or null | The number of partial images to generate. This parameter is used for streaming responses that return partial images. Value must be between 0 and 3. When set to 0, the response will be a single image sent in one streaming event. Not supported for now. | |
| 68 | +| quality | ❌ | ✅ | string or null (default: auto) | Quality of the image that will be generated. Recognized OpenAI qualities, but currently not supported: auto, high, medium, low, hd, standard | |
| 69 | +| response_format | ⚠️ | ✅ | string or null (default: b64_json) | The format of the images in output. Recognized options: b64_json or url. Only b64_json is supported for now (default). | |
| 70 | +| user | ❌ | ✅ | string (optional) | A unique identifier representing your end-user that allows to detect abuse. Not supported for now. | |
| 71 | + |
| 72 | +### Parameters supported via `extra_body` field |
| 73 | + |
| 74 | +| Param | OpenVINO Model Server | OpenAI /images/generations API | Type | Description | |
| 75 | +|-----|----------|----------|---------|-----| |
| 76 | +| prompt_2 | ✅ | ❌ | string (optional) | Prompt 2 for models which have at least two text encoders (SDXL/SD3/FLUX). | |
| 77 | +| prompt_3 | ✅ | ❌ | string (optional) | Prompt 3 for models which have at least three text encoders (SD3). | |
| 78 | +| negative_prompt | ✅ | ❌ | string (optional) | Negative prompt for models which support negative prompt (SD/SDXL/SD3). | |
| 79 | +| negative_prompt_2 | ✅ | ❌ | string (optional) | Negative prompt 2 for models which support negative prompt (SDXL/SD3). | |
| 80 | +| negative_prompt_3 | ✅ | ❌ | string (optional) | Negative prompt 3 for models which support negative prompt (SD3). | |
| 81 | +| num_images_per_prompt | ✅ | ❌ | integer (default: `1`) | The same as base parameter `n`. | |
| 82 | +| num_inference_steps | ✅ | ❌ | integer (default: `50`) | Defines denoising iteration count. Higher values increase quality and generation time, lower values generate faster with less detail. | |
| 83 | +| guidance_scale | ✅ | ❌ | float (optional) | Guidance scale parameter which controls how model sticks to text embeddings generated by text encoders within a pipeline. Higher value of guidance scale moves image generation towards text embeddings, but resulting image will be less natural and more augmented. | |
| 84 | +| strength | ✅ | ❌ | float (optional) min: 0.0, max: 1.0 | Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a starting point and more noise is added the higher the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in `num_inference_steps`. A value of 1 essentially ignores `image` parameter. | |
| 85 | +| rng_seed | ✅ | ❌ | integer (optional) | Seed for random generator. | |
| 86 | +| max_sequence_length | ✅ | ❌ | integer (optional) | This parameters limits max sequence length for T5 encoder for SD3 and FLUX models. T5 tokenizer output is padded with pad tokens to 'max_sequence_length' within a pipeline. So, for better performance, you can specify this parameter to lower value to speed-up T5 encoder inference as well as inference of transformer denoising model. For optimal performance it can be set to a number of tokens for `prompt_3` / `negative_prompt_3` for SD3 or `prompt_2` for FLUX. | |
| 87 | + |
| 88 | +## Response |
| 89 | + |
| 90 | +| Param | OpenVINO Model Server | OpenAI /images/generations API | Type | Description | |
| 91 | +|-----|----------|----------|---------|-----| |
| 92 | +| data | ✅ | ✅ | array | A list of generated images. | |
| 93 | +| data.b64_json | ✅ | ✅ | string | The base64-encoded JSON of the generated image. | |
| 94 | +| data.url | ❌ | ✅ | string | The URL of the generated image if `response_format` is set to url. Unsupported for now. | |
| 95 | +| data.revised_prompt | ❌ | ✅ | string | The revised prompt that was used to generate the image. Unsupported for now. | |
| 96 | +| usage | ❌ | ✅ | dictionary | Info about assessed tokens. Unsupported for now. | |
| 97 | +| created | ❌ | ✅ | string | The Unix timestamp (in seconds) of when the image was created. Unsupported for now. | |
| 98 | + |
| 99 | + |
| 100 | +## Currently unsupported endpoints: |
| 101 | +- images/variations |
| 102 | + |
| 103 | +## Error handling |
| 104 | +Endpoint can raise an error related to incorrect request in the following conditions: |
| 105 | +- Incorrect format of any of the fields based on the schema |
| 106 | +- Tokenized prompt exceeds the maximum length of the model context. |
| 107 | +- Model does not support requested width and height |
| 108 | +- Administrator defined min/max parameter value requirements are not met |
| 109 | + |
| 110 | +## References |
| 111 | + |
| 112 | +[End to end demo with image edit endpoint](../demos/image_generation/README.md) |
| 113 | + |
| 114 | +[Code snippets](./clients_genai.md) |
0 commit comments