ValueError: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8 #22
Description
while akhbar/Meta-Llama-3.1-8B-Instruct-abliterated-GPTQ could not use,which hugging face show 500
Internal Error - We're working hard to fix this as soon as possible!,
i use John6666/Llama-3.1-8B-Lexi-Uncensored-V2-nf4
got this error
--- Starting Generation ---
Model: dev-nf4 (alternate), Res: 1024x1024, Steps: 28, CFG: 0.0, Seed: 102583931986661
Using standard sequence lengths: CLIP-L: 77, OpenCLIP: 150, T5: 256, Llama: 256
Skipping pipe.to(cuda:0) (CPU offload enabled).
Executing pipeline inference...
!!! ERROR during execution: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8
Traceback (most recent call last):
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-HiDream-Sampler\hidreamsampler.py", line 741, in generate
output_images = pipe(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-HiDream-Sampler\hi_diffusers\pipelines\hidream_image\pipeline_hidream_image.py", line 646, in call
) = self.encode_prompt(
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-HiDream-Sampler\hi_diffusers\pipelines\hidream_image\pipeline_hidream_image.py", line 331, in encode_prompt
prompt_embeds, pooled_prompt_embeds = self._encode_prompt(
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-HiDream-Sampler\hi_diffusers\pipelines\hidream_image\pipeline_hidream_image.py", line 480, in _encode_prompt
llama3_prompt_embeds = self._get_llama3_prompt_embeds(
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-HiDream-Sampler\hi_diffusers\pipelines\hidream_image\pipeline_hidream_image.py", line 278, in _get_llama3_prompt_embeds
outputs = self.text_encoder_4(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\accelerate\hooks.py", line 176, in new_forward
output = module._old_forward(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\transformers\utils\deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\transformers\models\llama\modeling_llama.py", line 842, in forward
outputs = self.model(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\transformers\models\llama\modeling_llama.py", line 594, in forward
layer_outputs = decoder_layer(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\transformers\models\llama\modeling_llama.py", line 336, in forward
hidden_states, self_attn_weights = self.self_attn(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\transformers\models\llama\modeling_llama.py", line 270, in forward
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\accelerate\hooks.py", line 171, in new_forward
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\accelerate\hooks.py", line 361, in pre_forward
set_module_tensor_to_device(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\accelerate\utils\modeling.py", line 357, in set_module_tensor_to_device
new_value = param_cls(new_value, requires_grad=old_value.requires_grad, **kwargs).to(device)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\bitsandbytes\nn\modules.py", line 331, in to
return self._quantize(device)
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\bitsandbytes\nn\modules.py", line 296, in _quantize
w_4bit, quant_state = bnb.functional.quantize_4bit(
File "H:\ComfyUI\ComfyUI\python\lib\site-packages\bitsandbytes\functional.py", line 1244, in quantize_4bit
raise ValueError(f"Blockwise quantization only supports 16/32-bit floats, but got {A.dtype}")
ValueError: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8
HiDream: ComfyUI is unloading all models, cleaning HiDream cache...
HiDream: Cleaning up all cached models...
Removing 'dev-nf4_alternate'...