Skip to content

Conversation

@kjh2159
Copy link

@kjh2159 kjh2159 commented May 20, 2025

1. Description

The current mllm cannot handle Qwen2.5 0.5B models.
For example, the existing mllm throws dummy words even though the model is FP32.
However, now QwenConfig can handle Qwen2.5 0.5B models.

Basically, this modified QwenConfig class and its member functions do not harm the compatibility of the existing functionalities.

Compatible Quantized Models: fp32, q4_0

2. Usage

The default Qwen model version of QwenConfig is 1.5.
To use Qwen2.5 0.5B models, you can add only "Qwen2.5" string to the arguments when constructing "QwenConfig" object.

// demo_qwen.cpp

// model_version can be "qwen2.5" or "Qwen2.5"
// default model_version is "Qwen1.5"
QWenConfig config(tokens_limit, model_billion, RoPEType::HFHUBROPE, model_version);

// ex)
// QWenConfig config(512, "0.5B", RoPEType::HFHUBROPE, "Qwen2.5");
# execution command example
./bin/demo_qwen \
    -m qwen2.5-0.5b-instruct-q4_0.mllm \
    -v vocab/qwen2.5_vocab.mllm \
    -e vocab/qwen2.5_merges.txt \
    -b 0.5B \
    -r Qwen2.5

3. Model Files

I utilized the quantization and converters that mllm team provides.
And, the files for deployment of qwen2.5-0.5b models are uploaded on my huggingface repository and you can download the file with curl command.

Click here for the huggingface repository.

Execute the following command to download it.

curl -L https://huggingface.co/kjh2159/Qwen2.5-0.5B-Instruct-MLLM/resolve/main/qwen2.5-0.5b-instruct_q4_0.mllm --output qwen2.5-0.5b-instruct_q4_0.mllm

kjh2159 and others added 2 commits May 20, 2025 06:38
Now QwenConfig can handle Qwen2.5 0.5B models
Compatibility: fp32, q4_0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant