rfc-02: Workflow JSON Widget Values Format

[christian-byrne]

Summary

This RFC proposes a new format for handling widget values in ComfyUI workflows by integrating them directly into the node inputs array instead of storing them in a separate widgets_values array. The new format improves type safety, maintainability, and self-documentation of workflows by making each widget value a named, typed input with explicit metadata. This change will require a version bump in the workflow schema from 1.0 to 2.0, but includes backward compatibility measures to ensure a smooth transition for existing workflows and custom nodes.

Proposed Change

{
  "id": 3,
  "type": "KSampler",
  "pos": [863, 186],
  "size": [315, 262],
  "flags": {},
  "order": 4,
  "mode": 0,
  "inputs": [
    { "name": "model", "type": "MODEL", "link": 1 },
    { "name": "positive", "type": "CONDITIONING", "link": 4 },
    { "name": "negative", "type": "CONDITIONING", "link": 6 },
    { "name": "latent_image", "type": "LATENT", "link": 2 },
-    {
-      "name": "seed",
-      "type": "INT",
-      "link": null,
-      "widget": {
-        "name": "seed"
-      }
-    }
+    { "name": "seed", "type": "INT", "value": 156680208700286, "link": null },
+    { "name": "denoise", "type": "FLOAT", "value": 1.0 },
+    { "name": "steps", "type": "INT", "value": 20 },
+    { "name": "cfg", "type": "FLOAT", "value": 8 },
+    { "name": "sampler_name", "type": "COMBO", "value": "euler" },
+    { "name": "scheduler", "type": "COMBO", "value": "normal" },
+    { "name": "denoise", "type": "FLOAT", "value": 1.0 }
  ],
  "outputs": [{ "name": "LATENT", "type": "LATENT", "links": [7], "slot_index": 0 }],
  "properties": {},
-  "widgets_values": [156680208700286, true, 20, 8, "euler", "normal", 1]
}

Links

• Full Rendered Proposal

[AustinMroz]

VHS has used a modified widgets_values format since November of 2023 to achieve similar benefits. Unifying the storage to inputs is better and has my full support. Accordingly, I am fully committed to updating VHS for compatibility as required.

However, it introduces a number of problems which VHS has already needed to implement workarounds for:

• The implementation for backwards compatibility does not provide a means to manually specify the order of widgets_values when an old format is loaded.
  • If a developer wants to maintain support for loading workflows in the old format, then the proposal requires that they never insert or remove widgets anyways.
    • Worse, a developer needing to respect both formats means you can't utilize the foundational benefits of the original array format either: More restrictions AND less flexibility.
  • VHS solves this by manually specifying the original ordering and requiring that all workflows use the new format.
    • VHS has these values hard coded in js. My prior proposal was that python node definitions have an optional attribute to both opt-in to the new format and optionally provide this conversion map, but this is incompatible with the all-or-nothing proposal here.
• The proposal does not provide a means to rename an input.
• The proposal provides no means of migrating the type of an input.

EDIT: The proposal includes a transition period after which exporting workflows in the old format will seemingly no longer be supported. This simplifies the number of edge cases for developers to address immensely.

[huchenlei]

• The proposal does not provide a means to rename an input.

Rename of input will be the same format as label field on input slot:

{
  "inputs": [
    {
      "name": "clip",
      "type": "CLIP",
      "link": 13,
      "label": "Custom CLIP label"
    }
  ]
}

[huchenlei]

• The proposal provides no means of migrating the type of an input.

I think this is a separate issue, as currently if we need to change type of an input, it would often require making a new node definition, and mark the legacy node as deprecated and point users to the new node.

[pingren]

Hey team, here are two quick thoughts:

1. Explicit "Widget vs. Socket"
   Instead of relying on link being null or undefined, consider using something like inputKind: 'widget' | 'socket'. It makes it more clear how each input should be rendered and also leaves room for future kinds if needed.

2. What Happens if Widget Values Are Missing in 2.0?
   Since value is optional in the Zod schema, I wonder if a node may define { "name": "some_param", "type": "INT" } with no value in edge cases, should we assume a default or show an error?

Everything else looks awesome! I like the direction this proposal is heading. As a developer working on third-party tools, I’ll start making plans to support these changes. Keep up the great work!

[huchenlei]

1. Explicit "Widget vs. Socket"

There is a plan to have replace the widget/socket conversion mechanism with something more intuitive to use, i.e. have the socket always available besides to the widget. At that stage, everything will be able to have link field, and the value field decides whether the input is a widget.

[huchenlei]

2. What Happens if Widget Values Are Missing in 2.0?
Since value is optional in the Zod schema, I wonder if a node may define { "name": "some_param", "type": "INT" } with no value in edge cases, should we assume a default or show an error?

Each widget type actually are supposed to have a default value if the node does not specify anything. For example, INT will be 0 and STRING will be empty string if the default value is not set in the node definition.