Method to save intermediate step images via the API?

[ddmmss92]

Is there a method to do this using the API for Automatic1111? I am specifically interested in the InstructPix2Pix endpoint but will welcome any feedback on the topic.

I have been able to successfully use the API for other general tasks but haven't figured out how to save the intermediate steps yet. I know there are Extensions to do so but I could not find a way to use these with the API (LMK if I missed something obvious here).

One idea I had was to save an image from 1 step, 2 steps, 3 steps, etc., but this is obviously very cumbersome/resource-intensive and I am also randomizing the seed in my instructpix2pix POSTs so the destination image would not be consistent.

I am not a developer so please forgive my naivete. Any help is greatly appreciated.

[Kilvoctu]

I haven't tested it myself, but I think you can pull images from /sdapi/v1/progress. This API endpoint is active during a generation and one of the fields is current_image. You can pull the image however frequently you want while the task is running.

[vladmandic]

you may need to use asyncio and aiohttp to get that working, take a look at my wrapper https://github.com/vladmandic/automatic/blob/master/cli/modules/sdapi.py

[ddmmss92]

Awesome, thank you @vladmandic. These modules seem much more apt for the goal here than the concurrent.futures module I was originally trying.

I tried using the wrapper that you kindly linked but was unfortunately unable to successfully use it as I kept encountering session errors, perhaps due to one of the following:

1. User error on my end (by far the most likely)
2. The parameter <?skip_current_image> being set to 'true' (errors not resolved when I tried with 'false').
3. Pointing to endpoints not available in the API (e.g. '/sdapi/v1/shutdown', although I could be misreading the documentation).

In trying to use asyncio and aiohttp otherwise (full code I am using to test below for reference), I find that even though I can get the image generation task and the progress check task to run concurrently, the progress check GETs will still wait for the image generation POST itself to complete before resuming their check every 1second. I know this because the output of the below code includes print timestamps of the progress checks — the progress checks begin and then there will be a few seconds missing while the image generation happens before the progress checks resume every 1second.

This approach isn't satisfactory for capturing every step of generation but was initially just intended to confirm that GETting progress updates during image generation was possible. I don't think that this is an issue of Python GIL given that saving intermediate images is possible via the web UI with extensions/scripts. I just haven't found the solution yet. I will continue working on this, please forgive the slow responses as I am learning this stuff as I go.

import io
import base64
from PIL import Image, PngImagePlugin
import time
import aiohttp
import asyncio

# set the URL root``
sdurl = "http://127.0.0.1:7860" 

# set headers for API config
paramheaders = {
    "accept": "application/json",
    "Content-Type": "application/json"
}

# specify the model checkpoint and progress update frequency for the API
modeldata = {
    "sd_model_checkpoint": "instruct-pix2pix-00-22000.safetensors [fbc31a67aa]",
    "show_progress_every_n_steps": int('1')
}

# send a POST request to set the API config
paramresponse = requests.post(url=f'{sdurl}/sdapi/v1/options', headers=paramheaders, json=modeldata)
print('instructpix2pix running and show_progress_every_n_steps is set to 1')

# set the image path for the input image
imglocation = Image.open("example local image.jpg")

# create a BytesIO object to hold the encoded image and then save it
img_bytes = io.BytesIO()
imglocation.save(img_bytes, format='PNG')

# encode the BytesIO object as base64
print('encoding local image...')
img_base64 = base64.b64encode(img_bytes.getvalue()).decode('utf-8')
print('done encoding local image...')

# set the parameters for the image generation process payload
ip2ppayload = {
  "init_images": [img_base64],
  "prompt": "turn it into a moutain",
  "denoising_strength": 1,
  "negative_prompt": "",
  "output_batches": 1,
  "sampler": "LMS Karras",
  "steps": 30,
  "seed": 0,
  "randomize_seed": True,
  "text_cfg": 8,
  "image_cfg": 0.9,
  "output_image_width": 512,
}

# define coroutine for image generation task
async def imagegentask():
    # wait for 3 seconds before starting the task to ensure that the progress gen task starts beforehand
    await asyncio.sleep(3)

    # send an HTTP POST request to generate an image from the input image and text prompt
    async with aiohttp.ClientSession() as gensession:
        print('starting to generate image at time ' + time.strftime('%H:%M:%S'))
        async with gensession.post(f'{sdurl}/instruct-pix2pix/img2img', json=ip2ppayload) as response:
            print('done generating image at time ' + time.strftime('%H:%M:%S'))
            ip2presponse = await response.json()

            # get the JSON response and decode each image data into an image, and save to a specified directory with metadata
            ip2pjsonpayload = ip2presponse
            for ip2pimages in ip2pjsonpayload['images']:
                sdimage = Image.open(io.BytesIO(base64.b64decode(ip2pimages.split(",",1)[0])))
                print('decoding generated image at time: ' + time.strftime('%H:%M:%S'))
                png_payload = {
                    "image": "data:image/png;base64," + ip2pimages
                }
                async with gensession.post(f'{sdurl}/sdapi/v1/png-info', json=png_payload) as pngresponse:
                    pnginfo = PngImagePlugin.PngInfo()
                    pnginfo.add_text("parameters", (await pngresponse.json()).get("info"))

                    sdimage.save('ip2p-test-output-denoising.1.1080.1.0.randomseed1.png', pnginfo=pnginfo)
            print('done saving generated image at time: ' + time.strftime('%H:%M:%S'))

# define the coroutine for the progress check task
async def progresstask():
    start_time = time.time()
    while time.time() - start_time < 8:
        async with aiohttp.ClientSession() as session:
            progressparams = {
                "skip_current_image": "false",
                "skip_current_text": "true"
            }
            progressheaders = {
                "accept": "application/json"
            }
            async with session.get(f'{sdurl}/sdapi/v1/progress', params=progressparams, headers=progressheaders) as progressresponse:
                print('progress update at time: ' + time.strftime('%H:%M:%S'))
                print(await progressresponse.json())
                await asyncio.sleep(1)

# run the event loop and tasks concurrently
async def main():
    # create task for image generation
    imagegen_task = asyncio.create_task(imagegentask())

    # create task for progress checking
    progresstask_task = asyncio.create_task(progresstask())

    # run tasks concurrently
    await asyncio.gather(imagegen_task, progresstask_task)

# start event loop
asyncio.run(main())

[vladmandic]

one nitpick, aiohttp session create is a non-trival task and prefers to maintain its connection pool - so initialize it once and reuse (and may as well set common url root there).
for example, remove async with aiohttp.ClientSession() as session:

so it becomes something like:

session = None
sd_url = "http://127.0.0.1:7860"

async def get_session():
    global session # pylint: disable=global-statement
    time = aiohttp.ClientTimeout(total = None, sock_connect = 10, sock_read = None) # default value is 5 minutes but needs longer when running training via api
    sess = aiohttp.ClientSession(timeout = time, base_url = sd_url)

async def progresstask():
    session = session if session is not None else await get_session()
    ...

[ddmmss92]

Thank you, I made that update :) For reference, I figured out the issue. I am embarrassed to admit it. The asyncio example that I shared above actually is functional, it is the instructpix2pix API endpoint itself that blocks GET requests while generating an image after receiving a POST request.

I tested the same code but sending to txt2img and img2img endpoints instead and it worked fine.

I reviewed img2img.py to see why the ip2p endpoint wasn't able to return GET requests while generating while the other API endpoints were capable but I have not made much progress. I think the GitHub etiquette is to highlight something like this in Issues so I will make another post there so people know.

[jinghuacao]

one nitpick, aiohttp session create is a non-trival task and prefers to maintain its connection pool - so initialize it once and reuse (and may as well set common url root there). for example, remove async with aiohttp.ClientSession() as session:

so it becomes something like:

session = None
sd_url = "http://127.0.0.1:7860"

async def get_session():
    global session # pylint: disable=global-statement
    time = aiohttp.ClientTimeout(total = None, sock_connect = 10, sock_read = None) # default value is 5 minutes but needs longer when running training via api
    sess = aiohttp.ClientSession(timeout = time, base_url = sd_url)

async def progresstask():
    session = session if session is not None else await get_session()
    ...

I want to let you know I stared at the code fruitlessly troubleshooting for 2 hours, before I asked ChatGPT and it told me

the get_session() function is not returning a value, and when it is used in the asynchronous function progresstask(), it results in None. so I quickly changed to my dumb coding style,

if not session:
    get_session()

[vladmandic]

Pointing to endpoints not available in the API (e.g. '/sdapi/v1/shutdown', although I could be misreading the documentation).

That specific endpoint is not (yet) in main branch, it in my fork only. All others are in main.

User error on my end (by far the most likely)

I took a quick look and it uses the right concept with two create_task joined with gather.
If this doesn't work, then blocking is inside server itself.

Check your WebUI -> Settings -> Live Previews - i believe they need to be enabled to allow refresh calls while txt2img is still running. But honestly, don't know if that unlocks concurrent API access.

[ddmmss92]

Thank you for taking a quick look, I appreciate it!

Unfortunately I have not been able to get any progress endpoint GET requests to return non-null values while the image generation process is happening. The progress checks always temporarily pause while the image gen job runs regardless of how often I send the progress checks.

I know that getting updates via the progress API endpoint is possible in general given that I see others are able to successfully use it (e.g. this guy). As such, this approach should work in theory, I just can't execute it in practice, so I will mark the answer above accordingly.

If anyone reads this in future and can point me in the right direction to successfully get progress updates during image generation please let me know.

[Kilvoctu]

This fork of my frontend has a working usage of /sdapi/v1/progress. Check out the changes in core/stablecog.py. It might be helpful.
Kilvoctu/aiyabot@0664d58

I tried the code and it does work; this is how it looked. The 2nd reply was updating in real-time with the stats and preview:

[ddmmss92]

Awesome, that is great to see it in action, nice work. Thank you for referring me to that example for reference.

For the sake of clarification, my progress GET requests do work on their own - e.g. if I generate an image via the UI and then run an API progress GET separately I receive the expected response that I can decode into an intermediate image. However, the issue that I have is that my API image generation POST requests are run within the same script as my progress GET requests. As such something is blocking the progress requests from firing off until the image generation is complete even when trying to run them asynchronously.

Your response reinvigorated me that I should be able to do this so I'll keep trying. If I ever solve this I will post here for posterity.

[garvitbandil]

following is how i am calling the API, can you suggest what changes should I make to print the time left in terminal:

url = "http://127.0.0.1:7860"
print(user_prompt)
print(color)
print(self.preSeed)
#self.preSeed=-1
payload = {
"prompt": user_prompt,
"steps": 5,
"seed": self.preSeed,
"batch_size": 1,
"steps": 45,
"cfg_scale": 7.5,
"width": self.widthText.text(),
"height": self.heightText.text(),
"tiling": True,
}

    response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)

    r = response.json()
    

    imageNumber = 1
    for i in r['images']:
        image = Image.open(io.BytesIO(base64.b64decode(i.split(",", 1)[0])))
        image.save(f'output{imageNumber}.png')  # Rename and save the image

[ddmmss92]

You need to GET updates from the /sdapi/v1/progress endpoint if I recall correctly. Right now you are just POSTing to the the txt2img endpoint. If you read the parameters of the progress endpoint there should be something like an ETA or time remaining, or perhaps you will need to calculate the time on your own, but somewhere in a progress-related endpoint there should be something to calculate or show this. I can't access this right now but my main recommendation to you would be to set up another API call to GET the progress.

[megalois]

you may need to use asyncio and aiohttp to get that working, take a look at my wrapper https://github.com/vladmandic/automatic/blob/master/cli/modules/sdapi.py

@vladmandic I think the link is broken.

Also, thanks for that answer. I am trying to run sdapi in an async way and this will be pretty useful.

[vladmandic]

Link was valid 3 month ago when I wrote this.
It's still in the same repo, under /cli/sdapi.py

[pnavitha]

Is there API support for sd_save_intermediate_images as I need to use it using APIs?