Making it easier for users to get started with CTranslate2 models

[jordimas]

Problem
Currently, users must manually convert models before they can get started. This extra step creates unnecessary friction and makes onboarding more difficult than it needs to be.

Tasks

• Create an organization in HuggingFace and add the CTranslate2 developers
• Create a pipeline that automatically uploads to HuggingFace all the models supported converted to CTranslate2
• Add huggingface_hub as an optional dependency to the Python bindings and provide a mechanism to load models from HuggingFace using a prefix like hf://

Example:
generator = ctranslate2.Generator("hf://ctranslate2/llama3-8b", device="cpu")

If somebody wants to get this task, it will be great

[jncraton]

Thank you for sharing these thoughts. Something like you've described sounds like it could be very beneficial for on-boarding new users! I do have a few ideas about the path forward.

Huggingface currently lists over 500 models that are finetunes of just Qwen 2.5 0.5b. Supporting everything compatible would be a very large task, and someone would likely have to manage determining which models are worth fully supporting via the CTranslate2 HF organization.

I'm not sure that we need new functionality within the CTranslate2 codebase in order to make getting started simpler. A few well-tested example models within a CTranslate2 HF organization may be enough. Here's what you suggested supporting:

generator = ctranslate2.Generator("hf://ctranslate2/llama3-8b", device="cpu")

The equivalent currently can be as simple as this:

from huggingface_hub import snapshot_download
generator = ctranslate2.Generator(snapshot_download("ctranslate2/llama3-8b", device="cpu"))

That's still quite simple, but it maintains a loose coupling between CTranslate2 and huggingface_hub. It has been noted that our examples need work (#1947). Our current chat example is 116 lines and requires learning how to convert a model before it can be used. I believe that something like the following (also runnable via colab) would be a more helpful initial example:

from huggingface_hub import snapshot_download, hf_hub_download
from tokenizers import Tokenizer
import ctranslate2

# Initialize generator and tokenizer
generator = ctranslate2.Generator(snapshot_download('jncraton/Qwen3-0.6B-ct2-int8'))
tokenizer = Tokenizer.from_file(hf_hub_download('jncraton/Qwen3-0.6B-ct2-int8', 'tokenizer.json'))

# Create Prompt and tokenize
prompt = ("<|im_start|>system\nAnswer concisely.<|im_end|>\n"
          "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n"
          "<|im_start|>assistant\n"
          "<think>\n\n</think>\n\n")
prompt_tokens = tokenizer.encode(prompt).tokens

# Generate
output_tokens = generator.generate_tokens([prompt_tokens], max_length=64)
tokenizer.decode([t.token_id for t in output_tokens])

[BBC-Esq]

I do have a namespace on HF of "ctranslate2-4you? ;-)

BTW I have a conversion program that I could put in the "examples" folder if that helps for the time being?

[jordimas]

Hello,

Thank you for sharing these thoughts. Something like you've described sounds like it could be very beneficial for on-boarding new users! I do have a few ideas about the path forward.

Huggingface currently lists over 500 models that are finetunes of just Qwen 2.5 0.5b. Supporting everything compatible would be a very large task, and someone would likely have to manage determining which models are worth fully supporting via the CTranslate2 HF organization.

Well, I was not thinking about all fine-tuned models obliviously, just the core models. For example, if you take Gemma 3, the official ones published by Google (it and non-it all sizes).

I'm not sure that we need new functionality within the CTranslate2 codebase in order to make getting started simpler. A few well-tested example models within a CTranslate2 HF organization may be enough. Here's what you suggested supporting:

Agreed, adding support for loading hf directly may be over-killing. This provides the same benefit with a simpler approach.

[jordimas]

I created https://huggingface.co/CTranslate2HQ since https://huggingface.co/CTranslate2 was taken.

My suggestion is that interested contributors be added as admins so we can maintain a single official repository as a collective effort with the most relevant models available and properly converted. I will like add this to the documentation and the github from page.

Share your HuggingFace username and I will be adding you

[jncraton]

I'm happy to be added if that is the desired path forward. My HF username is the same as my GitHub username.

I was not thinking about all fine-tuned models obliviously, just the core models. For example, if you take Gemma 3, the official ones published by Google (it and non-it all sizes).

That makes sense, and my fine-tune example was perhaps a little extreme, but it still leaves us to determine what counts as a "core" model. Not all new base models have new architectures. SmolLM and SmolLM2 are just Llama. all-MiniLM-L6-v2 is just BERT. Would we quantize and upload models like those when they are released, or is the goal here just to upload a model to test and demo a new architecture? For what its worth, onnx-community is similar to what you are proposing and currently hosts over 900 models.

[jordimas]

Hello @BBC-Esq,
Would you be interested in helping maintain a centralized HuggingFace repository for ready-to-use models? You’ve already done some great work that could be transferred over to give the project a strong starting point. Thanks!

[BBC-Esq]

@jordimas Sure, I'd be interested. My username is "ctranslate2-4you" on HF.

[BBC-Esq]

I've seen these huggingface "spaces" as well. I thought there was one that converted to GGUF and would tel you if a particular model isn't supported. Perhaps we could do something similar and link to it from the HQ and GitHub repository etc?

[jncraton]

@BBC-Esq onnx-community has a space like that:

https://huggingface.co/spaces/onnx-community/convert-to-onnx

[BBC-Esq]

Does the official person want to make the page and then I'll upload the conversion script? I'll use the ONNX one as a template basically a stream lit app.

[jordimas]

Hello @BBC-Esq @jncraton. I gave you access, you should be able to create any kind of artifact.
Proving a space to do conversions is really cool. And also have the most common models.

[BBC-Esq]

Sounds good. Did you wanna create the initial space and then one of us can work on the application that will actually do the conversion or how did you want to handle that?

[BBC-Esq]

Just uploaded the first model...I used this at the top of the "model card" in case people didn't know...causes it to show the model hierarchy, which makes it easier to find if someone is on the original Qwen3 repository looking specifically for quantized versions...

[jordimas]

Hello @BBC-Esq

Sounds good. Did you wanna create the initial space and then one of us can work on the application that will actually do the conversion or how did you want to handle that?

I’d say that whoever has the time can go ahead. You should have permission to create the space and make any necessary changes. Since the code is open and available on HF, I think that works well too.

[jordimas]

Just uploaded the first model...I used this at the top of the "model card" in case people didn't know...causes it to show the model hierarchy, which makes it easier to find if someone is on the original Qwen3 repository looking specifically for quantized versions...

I think that is fine to publish some initial models manually.

However, with the numbers of models that we will need to publish it will good to have an pipeline that converts the models and uploads to HF. I think that also to have predictable name convention it is important and also some metadata in the repo with the versions of the tools, date, etc.

There are a few options but this pipeline can be stored in a space where we store the code, I will suggest to do it in Python.