Efficient batch frame editing - videos.

[Symbiomatrix]

I went over some of the batch processing code last night, and it seems that with a small alteration, img2img's batch mode could load & convert frames en masse, by converting the "batch size" parameter in that mode to essentially "number of frames to be loaded per batch", and storing the correct filenames for the later save. Seems to be just an altered loop in img2img.process_batch. Fixing the seed to be the same across the batch is pretty easy too - the inner processing function can accept a list of seeds & subseeds as is, so it's just a matter of replicating the p.seed & p.subseed properties.
With a good chunk of spare vram, I think it would speed up frame processing a great deal (x2-x3 at least).

Haven't given much thought to inpainting and the myriad other features, so I'm not sure whether that'd break something terribly, but I'm wondering if anyone else has. I tried adapting controlnet's code to frames too, but it gets pretty convoluted with how it handles the preprocessed source image (currently just takes the first of init_images when creating the control image, then sends it off to the hook to do god knows what).

Here's the code for it so far:

def process_batch(p, input_dir, output_dir, inpaint_mask_dir, args):
    processing.fix_seed(p)
    if p.batch_size > 1:
        p.seed = [p.seed] * p.batch_size
        p.subseed = [p.subseed] * p.batch_size

    images = shared.listfiles(input_dir)

    is_inpaint_batch = False
    if inpaint_mask_dir:
        inpaint_masks = shared.listfiles(inpaint_mask_dir)
        is_inpaint_batch = len(inpaint_masks) > 0
    if is_inpaint_batch:
        print(f"\nInpaint batch is enabled. {len(inpaint_masks)} masks found.")

    print(f"Will process {len(images)} images, creating {p.n_iter * p.batch_size} new images for each.")

    save_normally = output_dir == ''

    p.do_not_save_grid = True
    p.do_not_save_samples = not save_normally

    state.job_count = len(images) * p.n_iter

    cnt = 0 # Batch counter, once filled or at the end will send to processing.
    limg = []
    limgfl = []
    for i, image in enumerate(images):
        state.job = f"{i+1} out of {len(images)}"
        if state.skipped:
            state.skipped = False

        if state.interrupted:
            break
        img = Image.open(image)
        # Use the EXIF orientation of photos taken by smartphones.
        img = ImageOps.exif_transpose(img)
        limg.append(img)
        limgfl.append(image)
        cnt = cnt + 1 
        if cnt >= p.batch_size or i == len(images) - 1:
            print("Collected images, running {}".format(i))
            p.init_images = limg # list of different images.
            
            if is_inpaint_batch:
                # try to find corresponding mask for an image using simple filename matching
                mask_image_path = os.path.join(inpaint_mask_dir, os.path.basename(image))
                # if not found use first one ("same mask for all images" use-case)
                if not mask_image_path in inpaint_masks:
                    mask_image_path = inpaint_masks[0]
                mask_image = Image.open(mask_image_path)
                p.image_mask = mask_image
                
            proc = modules.scripts.scripts_img2img.run(p, *args)
            if proc is None:
                proc = process_images(p)
    
            for n, processed_image in enumerate(proc.images):
                filename = os.path.basename(limgfl[n]) # Grab name from list.
    
                # Images are different now.
                # if n > 0:
                #     left, right = os.path.splitext(filename)
                #     filename = f"{left}-{n}{right}"
    
                if not save_normally:
                    os.makedirs(output_dir, exist_ok=True)
                    processed_image.save(os.path.join(output_dir, filename))
            
            # Reset counter, collect next batch.
            cnt = 0
            limg = []
            limgfl = []

[catboxanon]

I was thinking about this as well, this is definitely the expected way that batch size should behave when doing a batch img2img sequence. I would open a draft PR instead of a discussion though, you're more likely to get more eyes on it then and it would make discussing the code easier with the diff available.

[Symbiomatrix]

Please feel free to open a PR, if this is indeed a feasible idea. I'm a bit swamped and whilst I'd like to help, had to cut the investigation short; just thought I'd drop a few notes seeing as nobody else did, as far as I could tell.