(Advanced) Custom workflows

Workflow tab

This is a non-user friendly tab that allow users to run and inject their custom workflow on each process. However, if you don't understand much of these things, don't worry! There is an interesting option that we will discuss later in importing workflows from ComfyUI section.

Running custom workflows

In the custom workflow tab, users will be able to edit the custom workflow in API format (https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py#L14). This exact workflow will be sent to the server if you hit "Run this workflow" button at the bottom. The top worflow to dropdown list shows the mode you'll want to apply this workflow to for config purposes.

In order to make this process easy, txt2img, img2img and inpaint tabs all have a "Get workflow" button, that will load into the workflow tab, and in the corresponding config mode, the workflow from your current Comfy SD Krita plugin settings. Users will then be able to manually add nodes or edit their settings there.

Custom workflow image settings

When getting custom workflows, you will notice the image data has being pruned:

In order to keep the workflow easier to read, I implemented a way to store the original image data into the plugin's config. This is true for both LoadBase64Image and LoadBase64ImageMask nodes.

When running the workflow, that data will be restored to the node, meaning that the workflow will load the images that were present in the moment the "Get workflow" button was clicked, and not necessarily the current selection or imported images for controlnet, for example.

However, users are presented more options to load image data (case sensitive!):

• <<SelectedImage>> Will load the current selected image instead of the stored image in the workflow at the moment of running it.
• <<CurrentLayerAsMask>> This is for inpaint workflows only (workflow to: inpaint selected). This will set the selected contents of the current active layer as the inpaint mask.

A note on this: inpainting workflows behave differently. When you get the workflow from the inpaint tab, the current layer will be stored as mask, and the rest of the image will be the input image. For inpainting purposes, this is the kind of workflow you'll want. <<CurrentLayerAsMask>> setting is only enabled for inpainting workflows. For the rest of the modes (txt2img, img2img and upscale), it will be replaced by <<SelectedImage>>, meaning that, if you attempt to inpaint with them, the mask will become the alpha channel (or whatever channel you choose in the mask loader node).

Users are always able to use normal LoadImage nodes to load images from ComfUI's input directory in custom workflows too. Possibilities are limited by what ComfyUI offers you.

Injecting workflows

You'll see this checkbox appear on each of txt2img, img2img, inpainting, and upscaling tab. If checked, instead of generating the default workflow for your current settings, Comfy SD Krita plugin will run the specified custom workflow in the workflow tab for the corresponding mode (for txt2img, it will run the workflow that appears when you select workflow to: txt2img in workflow tab).

The difference of running the custom workflow this way, is that, even though you've loaded a custom workflow, plugin will dynamically be able to:

• Change the settings to those in the UI if the node ids match the ones used by the plugin. Additionally, you're be able to use <<Prompt>> and <<NegativePrompt>> placeholders to be replaced by the prompt and negative prompt wherever they are present in the workflow.
• Load enabled controlnet units with all the specified settings in the controlnet tab (depending on how the custom workflow is structured, might deactivate those alredy present).
• Load LoRAs found in prompt (depending on how the custom workflow is structured, might deactivate those already present).

For controlnet with injected workflows, make sure your custom workflow fulfills the following requirements:

• There is one KSampler with id of "3"
• There is one ClipTextEncode with an id of "6"
• There is another ClipTextEncode with an id of "7"

For LoRAs to work with injected workflow:

• There is a CheckpointLoaderSimple (or another node that outputs a MODEL) with an id of "4"
• There is one ClipTextEncode (or another node that receives a CLIP) with an id of "6"
• There is another ClipTextEncode (or another node that receives a CLIP) with an id of "7"
• There is one KSampler (or another node that receives a MODEL) with id of "3"

These are the nodes the newly generated nodes will be chained to. For the last three requirements, you can alternatively use "<<LastLoadedLora|{id-to-fall-back}>>" placeholder as input for nodes that can receive MODEL or CLIP and that way skip the ids requirement:

"<<LastLoadedLora|{fallback-id}>>" is a placeholder that you can use to reference the last LoRA's id in the chain. In case no LoRA has been loaded, it will use the id you put instead of {fallback-id}.

In case you want a node that outputs CLIP to be connected to the first LoRA in the chain, you must set its id to "ClipSetLastLayer" as its the id that the LoRA expects by default. In order to help you with the creation of custom workflows, here is a list of node ids that the plugin uses by default. Can be useful information.

Importing workflows from ComfyUI

Users can also import workflows directly from ComfyUI in API format. The steps are the following:

• Build your workflow in ComfyUI as you normally would.
• Go to ComfyUI's settings (gear icon beside the "Queue Size: ") and check enable dev mode options.
• You can now click on the new Save (API Format) button.

• Your custom built workflow is ready to be run or injected to the plugin. You can manually edit it if needed. (i.e for img2img, you'll want to use a LoadBase64Image with <<SelectedImage>> as input image. See Custom workflow image settings).

Updating injected workflow with your current settings in Krita

An useful way to instantly update your current workflow (for editing or debugging), is to inject your custom workflow, set your settings in the plugin, and then press "get workflow" with custom workflow enabled (injected) in the mode you're working on. This will generate a new workflow which is going to be the updated version of your original imported workflow + your ComfyUI SD plugin settings.

Placeholders

Placeholders are special strings you can use in your injected workflow that will be replaced by the corresponding data in your plugin settings at runtime. Make sure to write them between "" as when formatting, Json expects a string. As for now, we have 6 different placeholders that you can use, and have already been mentioned in this page:

• "<<PrunedImageData>>" will load the image data that was present in the input the moment you got the workflow from the "Get workflow" button. This placeholder can only be used in LoadBase64Image and LoadBase64ImageMask nodes.
• "<<SelectedImage>>" will load the data of the selected image in the canvas, base64 encoded. Can only be used in LoadBase64Image and LoadBase64ImageMask nodes.
• "<<CurrentLayerAsMask>>" will load the selected contents of the current active layer as a mask. Should only be used with LoadBase64ImageMask node, and is only available for inpainting workflows.
• "<<LastLoadedLora|{fallback-id}>>" wherever this placeholder is found in the workflow, will be replaced by the id of the last LoadLora node in the generated workflow. Useful to make custom connections using loras as the input model (knowing that loras will be loaded beforehand). If no LoRA was loaded, it will use the id you put in {fallback-id}. Example, <<LastLoadedLora|4>> will put a "4" if no LoadLora was found in the workflow. The intended use of {fallback-id} is to put the ids that were originally in that input and that way prevent validation errors if no LoRA was loaded.
• "<<Prompt>>" placeholder that will be replaced by your prompt. Useful if you want to have more than one ClipTextEncode nodes with the same prompt without having to manually edit it in the workflow.
• "<<NegativePrompt>>" same as above but with the negative prompt.

Pre-crafted custom workflows

You can check some custom workflows ready for injection here. You can use them as examples of what you can do, or directly inject them if you find them useful.