Cant select nf4 Models?

[GoudaCouda]

When trying to load the sampler it doesn't give me any options to select nf4.

[GoudaCouda]

Looks like an issue with auto-gptq because Its impossible for me to install it on cu124. There seems to be a pr ready for this on the mian branch

[Excidos]

I just changed the code a bit and now it uses my NF4 model that was downloaded via the original Nodes, can provide the changes if people want.

[Granddyser]

I just changed the code a bit and now it uses my NF4 model that was downloaded via the original Nodes, can provide the changes if people want.

Yes please !

[Excidos]

https://drive.google.com/file/d/1QAas0fmOq4vgluiPIj6k-HfrI2jTHc6l/view?usp=drive_link

Heres a link to the edited hidreamssampler.py just replace it in the custom_node

otherwise if you want to do it yourself

# Fix for 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes' error

This PR addresses an error with the current version of transformers library where BitsAndBytesConfig no longer has the get_loading_attributes method.

## Changes

1. Modified the model loading process to handle this error with fallback approaches
2. Updated model loading dependencies from `auto_gptq` to `gptqmodel`

### Implementation

Replace the section in hidreamsampler.py from:
```python
print(f"[1c] Loading Text Encoder: {llama_model_name}... (May download files)"); text_encoder = LlamaForCausalLM.from_pretrained(llama_model_name, **text_encoder_load_kwargs)

To:

print(f"[1c] Loading Text Encoder: {llama_model_name}... (May download files)")
try:
    # First attempt - original approach
    text_encoder = LlamaForCausalLM.from_pretrained(llama_model_name, **text_encoder_load_kwargs)
except AttributeError as e:
    if "get_loading_attributes" in str(e):
        print(f"     ⚠️ Caught BitsAndBytesConfig error: {e}")
        print(f"     Trying alternative loading method...")
        
        # Remove the config key if it exists, as it might be causing issues
        if "config" in text_encoder_load_kwargs:
            config_backup = text_encoder_load_kwargs.pop("config")
            print(f"     Temporarily removed custom config")
        else:
            config_backup = None
            
        # Add output_hidden_states=True which is in the working version
        text_encoder_load_kwargs["output_hidden_states"] = True
        
        # Try loading without quantization config for NF4 models
        if is_nf4:
            print(f"     Using alternative loading for NF4 model")
            try:
                from transformers import AutoModelForCausalLM
                # Try to load using AutoModelForCausalLM which might handle things differently
                text_encoder = AutoModelForCausalLM.from_pretrained(
                    llama_model_name, 
                    **text_encoder_load_kwargs
                )
            except Exception as e2:
                print(f"     ⚠️ Alternative loading also failed: {e2}")
                print(f"     Trying one more approach with minimal args...")
                # One more fallback with minimal arguments
                text_encoder = LlamaForCausalLM.from_pretrained(
                    llama_model_name,
                    torch_dtype=model_dtype,
                    low_cpu_mem_usage=True
                )
        else:
            # For non-NF4, retry with the most basic options
            print(f"     Retrying with basic options")
            text_encoder = LlamaForCausalLM.from_pretrained(
                llama_model_name,
                torch_dtype=model_dtype,
                low_cpu_mem_usage=True
            )
    else:
        # Different AttributeError, just re-raise
        raise
if "device_map" not in text_encoder_load_kwargs: 
    print("     Moving text encoder to CUDA...")
    text_encoder.to("cuda")
    
step1_mem = torch.cuda.memory_allocated() / 1024**2 if torch.cuda.is_available() else 0
print(f"✅ Text encoder loaded! (VRAM: {step1_mem:.2f} MB)")

Also replace dependency:

import auto_gptq

With:

import gptqmodel

[Granddyser]

https://drive.google.com/file/d/1QAas0fmOq4vgluiPIj6k-HfrI2jTHc6l/view?usp=drive_link

Heres a link to the edited hidreamssampler.py just replace it in the custom_node

otherwise if you want to do it yourself

Fix for 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes' error

This PR addresses an error with the current version of transformers library where BitsAndBytesConfig no longer has the get_loading_attributes method.

Changes

1. Modified the model loading process to handle this error with fallback approaches
2. Updated model loading dependencies from auto_gptq to gptqmodel

Implementation

Replace the section in hidreamsampler.py from:

print(f"[1c] Loading Text Encoder: {llama_model_name}... (May download files)"); text_encoder = LlamaForCausalLM.from_pretrained(llama_model_name, **text_encoder_load_kwargs)
To:

print(f"[1c] Loading Text Encoder: {llama_model_name}... (May download files)")
try:
    # First attempt - original approach
    text_encoder = LlamaForCausalLM.from_pretrained(llama_model_name, **text_encoder_load_kwargs)
except AttributeError as e:
    if "get_loading_attributes" in str(e):
        print(f"     ⚠️ Caught BitsAndBytesConfig error: {e}")
        print(f"     Trying alternative loading method...")
        
        # Remove the config key if it exists, as it might be causing issues
        if "config" in text_encoder_load_kwargs:
            config_backup = text_encoder_load_kwargs.pop("config")
            print(f"     Temporarily removed custom config")
        else:
            config_backup = None
            
        # Add output_hidden_states=True which is in the working version
        text_encoder_load_kwargs["output_hidden_states"] = True
        
        # Try loading without quantization config for NF4 models
        if is_nf4:
            print(f"     Using alternative loading for NF4 model")
            try:
                from transformers import AutoModelForCausalLM
                # Try to load using AutoModelForCausalLM which might handle things differently
                text_encoder = AutoModelForCausalLM.from_pretrained(
                    llama_model_name, 
                    **text_encoder_load_kwargs
                )
            except Exception as e2:
                print(f"     ⚠️ Alternative loading also failed: {e2}")
                print(f"     Trying one more approach with minimal args...")
                # One more fallback with minimal arguments
                text_encoder = LlamaForCausalLM.from_pretrained(
                    llama_model_name,
                    torch_dtype=model_dtype,
                    low_cpu_mem_usage=True
                )
        else:
            # For non-NF4, retry with the most basic options
            print(f"     Retrying with basic options")
            text_encoder = LlamaForCausalLM.from_pretrained(
                llama_model_name,
                torch_dtype=model_dtype,
                low_cpu_mem_usage=True
            )
    else:
        # Different AttributeError, just re-raise
        raise
if "device_map" not in text_encoder_load_kwargs: 
    print("     Moving text encoder to CUDA...")
    text_encoder.to("cuda")
    
step1_mem = torch.cuda.memory_allocated() / 1024**2 if torch.cuda.is_available() else 0
print(f"✅ Text encoder loaded! (VRAM: {step1_mem:.2f} MB)")
Also replace dependency:

import auto_gptq
With:

import gptqmodel

Edit: Cannot access because of permission
Many thanks !!! Is it possible to use the uncensored LLM with it ?

[badjano]

sorry for spamming, but that didn't work

[badjano]

it seems to be some problem with gptqmodel installation

F:\ComfyUI2\.venv\Scripts>pip install gptqmodel --no-cache-dir
WARNING: Ignoring invalid distribution ~angsmith (C:\Python311\Lib\site-packages)
WARNING: Ignoring invalid distribution ~ip (C:\Python311\Lib\site-packages)
WARNING: Ignoring invalid distribution ~ttpx (C:\Python311\Lib\site-packages)
Collecting gptqmodel
  Downloading gptqmodel-2.2.0.tar.gz (284 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\William Queen\AppData\Local\Temp\pip-install-oofjav5_\gptqmodel_5c3a0a225d6e45dfbfa14891469da8b8\setup.py", line 24, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[badjano]

I give up

BTW, I'm on windows 11, so I guess this repo must be only working for linux users

[MijnSpam]

For those with no option to select NF4, make sure you don't have the Comfy Manager HiDream installed in custom-nodes (all lowercase letters) This Sample has Comfy as with a capital to start with as customnodes. When you have both then only the manager version is shown.
I had that issue and fixed it by uninstalling the Mananger version. (after that I still have other issues :P )

[sidework]

For folks having problems with GPTQModel (it wont build the wheel on 3.12), you can download a precompiled one at https://github.com/ModelCloud/GPTQModel/releases/download/v2.2.0/gptqmodel-2.2.0+cu124torch2.5-cp312-cp312-linux_x86_64.whl and install that.