Skip to content

fix(merge_model): use torch_dtype for MLCDVisionModel and remove invalid device_map from CLIPImageProcessor#104

Open
inigopm wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
inigopm:fix/merge-model-from-pretrained-params
Open

fix(merge_model): use torch_dtype for MLCDVisionModel and remove invalid device_map from CLIPImageProcessor#104
inigopm wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
inigopm:fix/merge-model-from-pretrained-params

Conversation

@inigopm
Copy link
Copy Markdown

@inigopm inigopm commented Mar 2, 2026

Problem

Two bugs in validate_vit_consistency() introduced by #52 and only partially addressed in #88:

  1. MLCDVisionModel.from_pretrained — missing torch_dtype=torch.float32. The original code used torch_dtype= correctly; refactor(merge_model): Enhanced file implementation robustness #52 changed it to dtype= (silently ignored by from_pretrained); Update merge_model.py #88 removed the param entirely, losing the explicit float32 cast.

  2. CLIPImageProcessor.from_pretrained — still has device_map=... after Update merge_model.py #88. CLIPImageProcessor is a ProcessorMixin, not a PreTrainedModel; device_map is not a valid argument and raises TypeError at runtime.

Fix

# before
rice_model = MLCDVisionModel.from_pretrained(vit_path, device_map={"": f"cuda:{CUDA_DEVICE}"})
processor  = CLIPImageProcessor.from_pretrained(vit_path, device_map={"": f"cuda:{CUDA_DEVICE}"}, use_fast=True)

# after
rice_model = MLCDVisionModel.from_pretrained(vit_path, device_map={"": f"cuda:{CUDA_DEVICE}"}, torch_dtype=torch.float32)
processor  = CLIPImageProcessor.from_pretrained(vit_path, use_fast=True)

…lid device_map from CLIPImageProcessor

CLIPImageProcessor is a processor, not a model — device_map is not a valid
argument and raises TypeError at runtime. MLCDVisionModel.from_pretrained
should use torch_dtype= (not dtype=) to explicitly set float32.

Regression introduced in EvolvingLMMs-Lab#52, partially addressed in EvolvingLMMs-Lab#88.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant