Add support for Higgsv2 + Autoregressive Generation #9736

yousef-rafat · 2025-09-05T20:49:23Z

into yousef-higgsv2

Kosinkadink

Thank you for the PR, and sorry it took so long to review! Comfy and I took a look today. There are some comments added, but here is a summary + extras:

CUDA Graph stuff should be removed from the code if possible.
comfy would prefer that the caches from transformers.cache_utils not be used, as he wants to have as little dependency on transformers as possible.
Check if the llama tokenizer .json could be reused for the higgsv2 tokenizer since they might be identical.
Torch over numpy wherever possible

While testing after creating the combined checkpoint file, I found a bug - if you try to run a workflow a second time by incrementing the seed, the Autoregressive Generation node does things for a bit but then ultimately throws this error:

!!! Exception during processing !!! 'StaticCache' object has no attribute 'layers'
Traceback (most recent call last):
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 277, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\nodes.py", line 1588, in generate
    return (auto_sample(self, model, input_ids, max_new_length, min_new_length, top_k, top_p, temperature, do_sample, seed = seed),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\autoregressive_sampling.py", line 678, in auto_sample
    samples = node._cached_autoregressive_sampler.generate(main_input_ids, max_new_length, min_new_length, top_k, top_p, temperature, do_sample, seed=seed, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Kosinkadink\ComfyUI\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\autoregressive_sampling.py", line 393, in generate
    result = self.model._sample(
             ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\ldm\higgsv2\model.py", line 1115, in _sample
    past_key_values, self.current_past_key_values_bucket = self._prepare_kv_cache(
                                                           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\ldm\higgsv2\model.py", line 1018, in _prepare_kv_cache
    self._copy_kv_cache(
  File "C:\Users\Kosinkadink\ComfyUI\comfy\ldm\higgsv2\model.py", line 983, in _copy_kv_cache
    from_layer = from_cache.layers[i]
                 ^^^^^^^^^^^^^^^^^
AttributeError: 'StaticCache' object has no attribute 'layers'```

Let me know if you have any questions/comments!

Kosinkadink · 2025-10-23T23:04:42Z

comfy/ldm/higgsv2/cuda_graph_runner.py

+
+_NUM_WARMUP_ITERS = 2
+
+class CUDAGraphRunner(nn.Module):


Comfy wants all CUDA graph stuff removed from this PR - unless there is a clear performance benefit. If the torch.cuda.synchronize call is needed, something may be wrong.

There's a noticeable and clear performance boost from CUDA graphs from my tests. You can see that by forcibly enabling/disabling them in the init of the AutoRegressiveGeneration class.

The torch.cuda.synchronize calls were in the original implementation: https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/cuda_graph_runner.py
I think I could remove them.

Gotcha, comfy says that we can eventually just make CUDA graphs a general comfy feature, so we shouldn't implement this for a specific model right now

Kosinkadink · 2025-10-23T23:07:36Z

comfy/autoregressive_sampling.py

+import warnings
+from enum import Enum
+from dataclasses import dataclass, fields
+from transformers.cache_utils import StaticCache, DynamicCache, Cache


Comfy would prefer if cache classes were not imported from transformers, so these likely need to use either some existing ComfyUI cache class or be rewritten.

Kosinkadink · 2025-10-23T23:14:48Z

comfy/ldm/higgsv2/loudness.py

+        return data
+
+    def apply_filter(self, data: torch.Tensor):
+        if data.is_cuda or self.use_fir:


There shouldnt be separate code paths for CPU/GPU, if possible.

The FIR filter does an FFT convolution, which benefits much from the gpu compared to a sequential algorithm like the IIR that benefits more from the cpu

How big is the difference?

I have run some tests, and it seems that FIR does well on both gpu and cpu compared to IIR, so I will stick with that
fir-vs-iir-performance_.ipynb

Kosinkadink · 2025-10-23T23:15:38Z

comfy/ldm/higgsv2/loudness.py

+    def generate_coefficients(self):
+
+        A  = 10**(self.G/40.0)
+        w0 = 2.0 * np.pi * (self.fc / self.rate)


The numpy code should be replaced with torch wherever possible

Kosinkadink · 2025-10-23T23:32:17Z

Another thing - when you create a checkpoint for these PRs, could you upload those to huggingface to make it simple to test?

Kosinkadink · 2025-10-25T22:02:25Z

I'll review your changes in the next day or so!

nodes.py

comfyanonymous · 2025-10-29T02:30:27Z

comfy/text_encoders/llama.py

-    return q_embed.to(org_dtype), k_embed.to(org_dtype)
+    return q_embed.to(org_dtype), k_embed.to(org_dtype), sin, cos
+
+class LlamaRoPE(nn.Module):


Can you move this out of this file?

To remove it or put it into another specific file?

comfyanonymous · 2025-10-29T02:30:41Z

comfy/text_encoders/llama.py

    mlp_activation = "silu"
+    qkv_bias: bool = False
+    rope_type: str = "llama3"
+    rope_scaling: dict = field(


Is this actually needed?

they are used in the llama3 rope calculations:
https://github.com/huggingface/transformers/blob/a43b36cf802f00616800e0bd4d748679236123ee/src/transformers/modeling_rope_utils.py#L532

Kosinkadink · 2025-11-14T03:28:33Z

Encountered an error trying to run:

!!! Exception during processing !!! 'str' object has no attribute 'shape'
Traceback (most recent call last):
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "C:\Users\Kosinkadink\ComfyUI\execution.py", line 277, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy_extras\nodes_autoregressive.py", line 48, in decode
    return clip.cond_stage_model.decode_tokens(tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\text_encoders\higgsv2.py", line 70, in decode_tokens
    vq_code = revert_delay_pattern_vectorized(audio).clip(0, self.audio_codebook_size - 1)[:, 1:-1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Kosinkadink\ComfyUI\comfy\text_encoders\higgsv2.py", line 14, in revert_delay_pattern_vectorized
    num_codebooks, total_len = data.shape
                               ^^^^^^^^^^
AttributeError: 'str' object has no attribute 'shape'

Workflow:
higgsv2_bug.json

Kosinkadink · 2025-11-14T03:32:37Z

The audio_tokens in this function is a dictionary instead of an iterable of tensors, I assume something was not pulled out properly?

Kosinkadink · 2025-11-19T21:28:51Z

Based on our conversation on slack, the tokens need to go through the autoregressive sampler before becoming useful. Because the output completely change form, there should be a different type outputted from the sampler than the input, otherwise users would be able to make the same mistake very easily and plug things in where they don't belong. Not sure if 'ENCODED_TOKENS' would be the best name for it, but something like that. If there is another way to do this, do let me know and we can review.

into yousef-higgsv2

init

254622d

yousef-rafat requested review from Kosinkadink, christian-byrne, comfyanonymous, guill, ltdrdata, pythongosssss, robinjhuang, webfiltered and yoland68 as code owners September 5, 2025 20:49

yousef-rafat added 3 commits September 5, 2025 23:53

Merge branch 'master' into yousef-higgsv2

1cff9b8

removed test files

df4b6a2

Merge branch 'yousef-higgsv2' of https://github.com/yousef-rafat/ComfyUI

6e9335d

into yousef-higgsv2

yousef-rafat changed the title ~~Add support to Higgsv2 + Autoregressive Generation~~ Add support for Higgsv2 + Autoregressive Generation Sep 5, 2025

yousef-rafat added 12 commits September 6, 2025 01:17

styling fixes

57c15f9

additional styling

f8d4891

.

233e441

bug fixes + added some features

6412422

Merge branch 'master' into yousef-higgsv2

5191fb2

final

2ac8999

Merge branch 'yousef-higgsv2' of https://github.com/yousef-rafat/ComfyUI

fee1e57

into yousef-higgsv2

Merge branch 'master' into yousef-higgsv2

86df359

Merge branch 'master' into yousef-higgsv2

cf18a06

Update supported_models.py

b17463d

Merge branch 'master' into yousef-higgsv2

b583c39

Merge branch 'master' into yousef-higgsv2

acdb10a

Kosinkadink added the Good PR This PR looks good to go, it needs comfy's final review. label Sep 18, 2025

Merge branch 'master' into yousef-higgsv2

ca7ecae

Kosinkadink added the Core Core team dependency label Sep 30, 2025

Merge branch 'master' into yousef-higgsv2

07cfed8

Kosinkadink reviewed Oct 23, 2025

View reviewed changes

yousef-rafat added 6 commits October 24, 2025 18:09

removed cuda graphs + changes to loudness.py

203849f

removed unused import

b80cbec

forgot cudagraph import in higgsv2 model.py

faa8667

.

01b3252

cache updates

76fdb4b

ruff checks

907ed37

Kosinkadink reviewed Oct 29, 2025

View reviewed changes

nodes.py Show resolved Hide resolved

comfyanonymous reviewed Oct 29, 2025

View reviewed changes

moved autoregressive nodes into an extra file

90e7674

yousef-rafat requested review from Kosinkadink and comfyanonymous October 30, 2025 13:57

Merge branch 'master' into yousef-higgsv2

01e93c7

yousef-rafat added 4 commits November 20, 2025 18:18

changed return type to generated_tokens

470bd9e

Merge branch 'master' into yousef-higgsv2

21ac52a

updated name to generated_tokens

c7851ae

Merge branch 'yousef-higgsv2' of https://github.com/yousef-rafat/ComfyUI

4ece4ee

into yousef-higgsv2

Add support for Higgsv2 + Autoregressive Generation #9736

Are you sure you want to change the base?

Add support for Higgsv2 + Autoregressive Generation #9736

Conversation

yousef-rafat commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kosinkadink left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kosinkadink commented Oct 23, 2025

Uh oh!

Kosinkadink commented Oct 25, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kosinkadink commented Nov 14, 2025

Uh oh!

Kosinkadink commented Nov 14, 2025

Uh oh!

Kosinkadink commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yousef-rafat commented Sep 5, 2025 •

edited

Loading

Kosinkadink commented Nov 19, 2025 •

edited

Loading