Development Discussion

[Kosinkadink]

We can use this discussion thread to share plans for nodes and collaborate.

[AustinMroz]

The funky voodoo merge has been pushed to migrated_code. I've since done some cleanup and scaffolding that I'm testing before I push that as well.

[bluesblood]

IMPORT FAILED ,why?

[Kosinkadink]

You'll need to provide some more details. Look in your console, it should give a full error stack for the import failed.

[AustinMroz]

I've been working on a number of feature branches and thought it best to provide an update on each here ot get some insight on which are worth opening pull requests for. All are mostly functional, but would need some more extensive debugging.

Format Widgets

This provides a means for video_formats to define additional options which are exposed in the webui as widgets. It cleans up the inclusion of some arguements like crf being shown for the animated image formats that they don't apply to and allows for the creation of formats that use parameters incompatible with crf, like fixed bit rate. It also allows for default values for options like crf to be set per format.

widgets.webm

Generated Video Previews

This introduces a new html endpoint that renders video previews from the given file on disk.
With this update, previews in the Load Video (Upload) node are accurate to the values entered for settings like force_size, frame_load_cap, and skip_first_frames, and videos are automatically downsized based on the size of the node.
Performance from testing is better than expected due to the small size of nodes on screen.

preview.webm

Currently, this functionality is implemented with ffmpeg as I have not yet found a way to make with work with opencv

Support for Increased Color Depth

As images internally store each pixel as 3 32bit floats, there had been talk in the past about using this extra information to produce output images with higher color depth. I wrote code to instead pass color values to ffmpeg as a series of shorts instead of bytes with the intent of providing higher quality output when 10bit pix_fmts were selected, but was also able to quickly draft a video_format to output 16 bit pngs for testing with.

Results have not been promising. Tests were performed by subtracting 8 bit png output from 16 bit png to observe what data is stored in the 8 least significant bits, but the resulting output seems to closely mirror a desaturated version of the image rather than a provider of useful information. Here is a comparison of an image output, the image desaturated, and the data provided in the lower 8 bits.

[AustinMroz]

A while back, there was a request for the path based nodes to provide suggestions. This was a fairly significant undertaking, but I finally have a functioning implementation.

completion.webm

[ryanontheinside]

Thank you so much for this and all of your hard work!

[exnex]

Hello, I was wondering if it would be possible to add a node or an output option for the audio of the video file to be compatible with this node's input?

https://github.com/a1lazydog/ComfyUI-AudioScheduler

It requires wav or mp3 loaded to input, so sending the video audio currently doesn't work. Currently, in order to sync the video audio to these nodes i need to extract the video audio first which is very inefficient. Unless there's another node pack for audio scheduling that is compatible? Thank you. Also, my video combine node no longer displays the output, any idea whats up there? I think it might be my HDR monitor/windows issues.

[AustinMroz]

I'm not aware of any node sets that are compatible with the audio format used by VHS and I'd want to think things over for a while before committing to the maintenance of such interoperability long term.

However, here's a quick standalone plugin that you could place in custom_nodes:

VHS_AS_Interop.py

import numpy as np
import subprocess
import re
import importlib
ffmpeg_path = importlib.import_module('custom_nodes.ComfyUI-VideoHelperSuite.videohelpersuite.utils').ffmpeg_path
as_audio = importlib.import_module('custom_nodes.ComfyUI-AudioScheduler.audio')


class VHS_to_AS:
    @classmethod
    def INPUT_TYPES(s):
        return {"required": { "vhs_audio": ("VHS_AUDIO",)}}
    RETURN_TYPES = ("AUDIO",)
    RETURN_NAMES = ("as_audio",)
    FUNCTION = "convert_to_as"

    def convert_to_as(self, vhs_audio):
        res = subprocess.run([ffmpeg_path, "-i", "-", "-f", "s16le", "-"],
                             capture_output=True, check=True, input=vhs_audio())
        match = re.search(", (\\d+) Hz, (\\d+) channels,", res.stderr.decode("utf-8"))

        if not match:
            raise Exception("failed to parse channel/sample rate")
        ad = as_audio.AudioData.__new__(as_audio.AudioData)
        ad.sample_rate = int(match.group(1))
        ad.num_channels = int(match.group(2))
        ad.audio_data = np.frombuffer(res.stdout, dtype=np.int16)
        return (ad,)

format_dict = {"int16": "s16le", "int8": "s8", "int32": "s32le",
               "uint16": "u16le", "uint8": "u8", "uint32": "u32le",
               "float32": "f32le", "float16": "f16le"}
class AS_to_VHS:
    @classmethod
    def INPUT_TYPES(s):
        return {"required": { "as_audio": ("AUDIO",)}}
    RETURN_TYPES = ("VHS_AUDIO",)
    RETURN_NAMES = ("vhs_audio",)
    FUNCTION = "convert_to_vhs"

    def convert_to_vhs(self, as_audio):
        format_string = format_dict[str(as_audio.audio_data.dtype)]

        res = subprocess.run([ffmpeg_path, "-v", "error", "-f", format_string,
                              "-ar", str(as_audio.sample_rate), "-ac", str(as_audio.num_channels),
                              "-i", "-", "-f", "wav", "-"], input=as_audio.audio_data.tobytes(),
                             stdout=subprocess.PIPE, check=True).stdout
        return (lambda : res,)

NODE_CLASS_MAPPINGS = {
    "VHS_to_AS": VHS_to_AS,
    "AS_to_VHS": AS_to_VHS,
}

Regarding previews not displaying, what format are you outputting to? Some of the fancier formats (h265, ProRes) require the "VHS Advanced Previews" box to be checked, which would be my first guess.

[exnex]

Thank you, I forgot to reply when you posted this. it works perfect! And you were right, it was the previews option had somehow become unchecked.

[melanie0901]

How can I output the Mp4 format? I can only choose the GIF and Webp, please Help!!!!!