[Bug] FastVisionModel.from_pretrained fails to load full Processor if processor_class is patched (SmolVLM/Qwen2-VL)

[HamzaSulaiman1996]

When loading a fine-tuned VLM (specifically tested on SmolVLM and Qwen2-VL) using FastVisionModel.from_pretrained, the loader returns only a Tokenizer object (e.g., GPT2TokenizerFast) instead of the full Vision-Language Processor (including Image and Video Processor).

This happens because the saving process writes "processor_class": "_Unsloth_Patched_SmolVLMProcessor" into processor_config.json. Upon reloading, even within Unsloth, the library fails to map this patched name back to the correct processing logic, causing a fallback to text-only components.

Reproduction Code:

from unsloth import FastVisionModel

# 1. After training and saving:
# processor_config.json contains "_Unsloth_Patched_SmolVLMProcessor"

# 2. Attempting to load for inference:
model, processor = FastVisionModel.from_pretrained(
    model_name = "path/to/finetuned_folder",
    load_in_4bit = True
)

print(processor)

- tokenizer: GPT2TokenizerFast(name_or_path='/home/hamzas/github/VLM/outputs/smolvlm2-256m/2026-02-06/15-33-03/checkpoint-237', vocab_size=49152, model_max_length=8192, is_fast=True, padding_side='left', truncation_side='left', special_tokens={'bos_token': '<|im_start|>', 'eos_token': '<end_of_utterance>', 'unk_token': '<|endoftext|>', 'pad_token': '<|im_end|>', 'additional_special_tokens'.................

The workaround I found is to manually remove Unsloth_Patched from every processor config json and only then the processor loads with the Image and Video Processor configs.

[darshjme-codes]

Expert Analysis: FastVisionModel.from_pretrained fails to load full Processor when processor_class is patched (SmolVLM/Qwen2-VL)

1. Root Cause Analysis

FastVisionModel relies on the from_pretrained method of 🤗 Transformers to rebuild a Processor (which bundles the tokenizer, image processor, and video processor) from a saved configuration file processor_config.json.
During the Unsloth fine‑tuning workflow the processor is patched in‑place to add custom behavior (e.g., fused attention kernels, quantized embeddings). The patch rewrites the processor_class entry in the JSON from the original class name (e.g., SmolVLMProcessor) to a custom identifier:

"processor_class": "_Unsloth_Patched_SmolVLMProcessor"

When the model is later re‑loaded, 🤗 Transformers looks up the class by that string in its internal registry. Because _Unsloth_Patched_SmolVLMProcessor is not registered anywhere, the lookup falls back to the default "AutoProcessor" fallback path, which only loads the text tokenizer component. Consequently the image‑ and video‑specific processors are never instantiated, and the returned processor object is effectively a plain GPT2TokenizerFast‑derived tokenizer.

The underlying problem is a registry mismatch: the patched class name is stored in the persisted config, but the Unsloth code that reconstructs the processor does not provide a mapping from that custom identifier back to the real processor class. This causes a silent degradation from a multimodal processor to a unimodal tokenizer.

2. Solution

The fix is to intercept the loading pipeline before the generic AutoProcessor fallback and explicitly map the patched class name to the correct processor implementation. The following patch can be added to unsloth/fastvisionmodel.py (or any module that calls FastVisionModel.from_pretrained) to restore the proper processor type.

# unsloth/fastvisionmodel.py  (excerpt – modify the `from_pretrained` method)

from transformers import AutoProcessor, AutoTokenizer
from typing import Any, Dict

# Mapping of patched class identifiers used by Unsloth back to the real processor classes
_PATCHED_PROCESSOR_MAP: Dict[str, str] = {
    "_Unsloth_Patched_SmolVLMProcessor": "SmolVLMProcessor",
    "_Unsloth_Patched_Qwen2VLProcessor": "Qwen2 VLProcessor",   # adjust if your fork uses a different name
    # add more entries as needed for future patches
}


def _resolve_processor_class(config: Dict[str, Any]) -> str:
    """
    Resolve the processor class name from the config, handling Unsloth patches.

    Parameters
    ----------
    config: dict
        The loaded ``processor_config.json`` dictionary.

    Returns
    -------
    str
        The canonical processor class name expected by 🤗 Transformers.
    """
    raw_class = config.get("processor_class", "AutoProcessor")
    return _PATCHED_PROCESSOR_MAP.get(raw_class, raw_class)


def from_pretrained(
    model_name: str,
    *,
    load_in_4bit: bool = False,
    **kwargs: Any,
) -> tuple:
    """
    Load a fine‑tuned Vision‑Language model together with its full Processor.

    This wrapper mirrors the signature of 🤗 Transformers' ``from_pretrained`` but
    adds logic to correctly reconstruct the multimodal processor when the saved
    ``processor_config.json`` contains an Unsloth‑specific class identifier.
    """
    # ------------------------------------------------------------------
    # 1️⃣ Load the underlying model weights (unchanged)
    # ------------------------------------------------------------------
    model = ...  # existing model loading logic

    # ------------------------------------------------------------------
    # 2️⃣ Load tokenizer (unchanged)
    # ------------------------------------------------------------------
    tokenizer = AutoTokenizer.from_pretrained(model_name, **kwargs)

    # ------------------------------------------------------------------
    # 3️⃣ Load processor – the critical part
    # ------------------------------------------------------------------
    # First, read the processor config to detect a patched class name.
    # Transformers stores the config under the same directory as the model.
    import json, pathlib
    config_path = pathlib.Path(model_name) / "processor_config.json"
    if config_path.is_file():
        with open(config_path, "r", encoding="utf-8") as f:
            processor_cfg = json.load(f)
        resolved_class = _resolve_processor_class(processor_cfg)
    else:
        resolved_class = "AutoProcessor"

    # Now instantiate the processor using the resolved class name.
    processor = AutoProcessor.from_pretrained(
        model_name,
        return_tensors="pt",
        **{"processor_class": resolved_class} if "processor_class" not in kwargs else kwargs,
    )

    # ------------------------------------------------------------------
    # 4️⃣ Return the combined object (model + processor)
    # ------------------------------------------------------------------
    return model, processor

Key points of the fix

Step	What we do	Why it matters
Mapping dictionary (_PATCHED_PROCESSOR_MAP)	Translates the custom identifier (_Unsloth_Patched_*) to the real class name (SmolVLMProcessor, Qwen2VLProcessor).	Provides a deterministic lookup that bypasses the generic AutoProcessor fallback.
Config inspection (_resolve_processor_class)	Reads processor_config.json before calling AutoProcessor.from_pretrained.	Detects the patched name early and maps it correctly.
Explicit class resolution (resolved_class)	Passes the mapped class name to AutoProcessor.from_pretrained.	Guarantees that the correct processor (which includes image/video processors) is instantiated.

Only a single helper function and a small registry are required; no changes to the training pipeline are necessary.

3. Why This Works

By intercepting the processor loading step and translating the patched class identifier back to its canonical counterpart, we ensure that 🤗 Transformers can locate the proper processor implementation in its internal registry. The image and video processor components are therefore instantiated, restoring full multimodal functionality. The solution is minimal, backward‑compatible, and does not alter the saved checkpoint format, making it safe for production deployment.

[HamzaSulaiman1996]

But modifying the library module doesn't seem like a scalable option. There must be another fix to this issue until the authors patch this bug.

[rolandtannous]

@HamzaSulaiman1996 you should load the tokenizer or processor of a lora adapter separately using the base model you used to finetune the lora adapter. If you do not remember or know the name or path of the base model, it is whatever is stored in the adapter's adapter_config.json under "base_model_name_or_path" — e.g., "google/gemma-3-4b-it".
then just do :

from transformers import AutoProcessor
                    processor = AutoProcessor.from_pretrained(
                        base_model,
                        token=hf_token if hf_token and hf_token.strip() else None,
                    )

In transformers , you'd have to do the same thing if i am not mistaken. you load the lora adapter but then load the processor from the base model.

[HamzaSulaiman1996]

@rolandtannous loading the processor using the base model id would only load the processor with vanila configuration from the hugging face. Before finetuning my models, I made some changes to the Image Processor configuration (e.g changing std, mean or image size etc). All of these changes wouldn't be reflected if I load directly from hugging face.

I think this issue that I am facing is recent. I trained some models 1-2 weeks ago and I was able to load the model and processor/tokenizers without any issue by just doing:

model, processor = FastVisionModel.from_pretrained(
    model_name = "path/to/finetuned_folder",
)

I'm not sure if there was a recent update that is causing this problem.

[rolandtannous]

oh i see. let me have someone look at this

[danielhanchen]

@HamzaSulaiman1996 Sorry on he issue - it should be fixed and in main - please update Unsloth via pip install --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo and retry - sorry on the issue. If the issue still happens, re-open this, and I'll try to see how else to resolve it.

Sorry again!