Feature Request: LFM2-24B-A2B and LFM2-8B-A1B(lfm2moe architecture) support

[hardWorker254]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

What about adding support for LFM2-24B-A2B(https://huggingface.co/LiquidAI/LFM2-24B-A2B) and LFM2-8B-A1B(https://huggingface.co/LiquidAI/LFM2-8B-A1B)?

Motivation

Support for fast and lightweight MoE models

Possible Implementation

lfm2moe ggml-org/llama.cpp#16464

[magikRUKKOLA]

// offtopic

Can anyone please share what are the use-cases for these models. (Personally, I don't see any reason to run anything lower than 397B-A17B)

[hardWorker254]

// offtopic

Can anyone please share what are the use-cases for these models. (Personally, I don't see any reason to run anything lower than 397B-A17B)

I'll agree. Lightweight models are dumb relative to their larger counterparts. But if they were useless, then companies would continue to waste resources on this, right? First of all, these models are aimed at an audience with weak components (That's me. If a miracle happened, and my laptop Ryzen 7 8845HS would start providing normal performance (Perhaps the problem here is that I have only one 16 GB DDR5 RAM (I'm just greedy, and 16 GB is enough for me), and it runs at a frequency of 2800 Mhz. But if there was a problem with RAM, I wouldn't have such terrible performance on Radeon 780m with Vulkan), then I certainly wouldn't use such models. But they are still not bad, they have quite good results in writing code, mathematics and other areas. They can't be trusted with some complicated task, but I think it's possible to do something simple, like reorganizing the codebase using bash. I think that the main purpose of such models is to maintain good capabilities at very small sizes and to ensure confidentiality. This is just my opinion, and I would consider it quite normal if someone thinks that what I wrote doesn't make sense if the small models are just dumb. (Yeah, I used translator to wrote this text)

[as-omer]

// offtopic

Can anyone please share what are the use-cases for these models. (Personally, I don't see any reason to run anything lower than 397B-A17B)

I see no reason to run 397B-17A for thousands of concurrent summarization/extraction tasks. All in need is a smart-enough-but-not-too-smart, cheap and efficient LLM to do the job.

Also, real time (or near real time) processing.

So many use cases. I'd add IBM's Granite series to the request as well.

[magikRUKKOLA]

@as-omer

I see no reason to run 397B-17A for thousands of concurrent summarization/extraction tasks.

... extraction of what?

So many use cases.

Yeah, that's exactly the question -- what kind of tasks can be delegated to them?

[as-omer]

@magikRUKKOLA

think about any short text (<4 paragraphs) that you'd like. customer reviews, social media posts, short gov publications, whatever.

you can ask it for the topic, main themes, tagging the text using a predefined list of tags, extracting flags, emphasize certain aspects (text about a car model > focus on comfort after a long ride). extremely useful for better RAG and information retrieval pipelines, text classification etc. i'd be less inclined to give it long or free text generation tasks, but for json they're great for many use cases.

for me there was practically 0 difference between opus:latest to 8B1A for non-reasoning and 20B-40B MoE models for reasoning-based summarization/extraction tasks (as long as the task was self-contained, narrow and the input was short). and that thing runs locally on a potato gpu with >1ks pp and >100s tg. you can basically launch a swarm of these things at 0 cost.

personally used it on clients' internal documents and for personal social-media uses.

[FNsi]

I'd like to see something big with A1B, may work in swap, extremely long ctx like 10m... this A1B model looks interesting.

[FNsi]

// offtopic

Can anyone please share what are the use-cases for these models. (Personally, I don't see any reason to run anything lower than 397B-A17B)

I just knew how strong the 27B model is, so impressive, the speed is the only thing barring me...maybe you can try it as a replacement.