Skip to content

Conversation

@SunMarc
Copy link
Member

@SunMarc SunMarc commented Apr 17, 2025

What does this PR do?

Fixes tests/quantization/bnb/test_mixed_int8.py::SlowBnb8bitTests::test_moving_to_cpu_throws_warning due to this PR.
The issue is that since we used dispatch_model, when we load a 8-bit model, hooks will be attached to it (this is not the case for 4-bit). Due to that, pipeline_is_sequentially_offloaded is True -> is_offloaded also True. Thus, we didn't log the following :

            if (
                module.dtype == torch.float16
                and str(device) in ["cpu"]
                and not silence_dtype_warnings
                and not is_offloaded
            ):
                logger.warning(
                    "Pipelines loaded with `dtype=torch.float16` cannot run with `cpu` device. It"
                    " is not recommended to move them to `cpu` as running them will fail. Please make"
                    " sure to use an accelerator to run the pipeline in inference, due to the lack of"
                    " support for`float16` operations on this device in PyTorch. Please, remove the"
                    " `torch_dtype=torch.float16` argument, or use another device for inference."
                )

Traceback of the test before this PR

================================ short test summary info ================================
FAILED tests/quantization/bnb/test_mixed_int8.py::SlowBnb8bitTests::test_moving_to_cpu_throws_warning - assert 'Pipelines loaded with `dtype=torch.float16`' in "The module 'SD3Transformer2DModel' has been loaded in `bitsandbytes` 8bit and moving it to cpu via `.to()` is not supported. Module is still on cuda:0.\n"
 +  where "The module 'SD3Transformer2DModel' has been loaded in `bitsandbytes` 8bit and moving it to cpu via `.to()` is not supported. Module is still on cuda:0.\n" = captured: The module 'SD3Transformer2DModel' has been loaded in `bitsandbytes` 8bit and moving it to cpu via `.to()` is not supported. Module is still on cuda:0.\n\n.out

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@sayakpaul
Copy link
Member

Failing test is unrelated.

@sayakpaul sayakpaul merged commit bbd0c16 into main Apr 18, 2025
31 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants