Feature Request: BailingMoeV2 Support (Ling Lite 2.0)

[fizzAI]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add support for BailingMoeV2ForCausalLM models, like https://huggingface.co/inclusionAI/Ling-mini-2.0 and https://huggingface.co/inclusionAI/Ring-mini-2.0

Motivation

They are promising, small MoE models for lower end GPU and CPU users

Possible Implementation

While there isn't any existing implementation, they made available manual patches for vLLM and SGLang, as well as the Transformers custom code implementation in the HF repo, of course.

It looks like they mostly implemented it as an extension to the original BailingMoE and I think we should be able to do the same here

[theo77186]

there are not much differences for this one: attention has QK norm, MoE is like Deepseek V3 with sigmoid gating but without expert selection biases.

[foldl]

This is supported by chatllm.cpp.

Its experts selection is unique.

[warshanks]

It looks like this is being worked on in #16028 🎉

[im0qianqian]

Thanks for your attention; let’s look forward together to this PR #16028 being accepted by the llama.cpp.

[engrtipusultan]

@CISC Hoping to see the official support soon. Thank you for your hard work.