add smolvla to model-libraries.ts #1529
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -1637,4 +1637,29 @@ audio = model.autoencoder.decode(codes)[0].cpu() | |||||||||||||||||||||||
| torchaudio.save("sample.wav", audio, model.autoencoder.sampling_rate) | ||||||||||||||||||||||||
| `, | ||||||||||||||||||||||||
| ]; | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| export const smolvla = (model: ModelData): string[] => [ | ||||||||||||||||||||||||
| `# git clone https://github.com/huggingface/lerobot.git | ||||||||||||||||||||||||
| cd lerobot | ||||||||||||||||||||||||
| conda create -y -n lerobot python=3.10 | ||||||||||||||||||||||||
| conda activate lerobot | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| # Install ffmpeg (required for video processing) | ||||||||||||||||||||||||
| conda install ffmpeg=7.1.1 -c conda-forge | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| # Install LeRobot with the SmolVLA extra dependencies | ||||||||||||||||||||||||
| pip install -e ".[${model.id}]" | ||||||||||||||||||||||||
|
||||||||||||||||||||||||
| `# git clone https://github.com/huggingface/lerobot.git | |
| cd lerobot | |
| conda create -y -n lerobot python=3.10 | |
| conda activate lerobot | |
| # Install ffmpeg (required for video processing) | |
| conda install ffmpeg=7.1.1 -c conda-forge | |
| # Install LeRobot with the SmolVLA extra dependencies | |
| pip install -e ".[${model.id}]" | |
| `# !pip install "git+https://github.com/huggingface/lerobot.git#egg=lerobot[smolvla]" |
i.e. no need for editable mode, no need to download the repo just to install it, no need for conda instructions (users can figure that part by themselves, depending on what they prefer).
If users want more detailed installation guide, they can go to https://github.com/huggingface/lerobot?#installation
There was a problem hiding this comment.
nice! thanks!
danaaubakirova marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
We usually demonstrate simple inference snippets. Does that make sense for smolvla, or not really?
There was a problem hiding this comment.
agree with @pcuenca that an inference example feels more natural at this stage since if a user has pushed their model, it means it has already been trained (at least I expect).
FYI, export const smolvla = (model: ModelData): string[] => [ returns a list so it could return both a training and an inference examples.
There was a problem hiding this comment.
Yes, I see what you mean. However, unfortunately we cannot use smolvla_base for inference directly, the only use case for smolvla_base for now is fine-tuning.
As it is not meant to perform zero-shot and will cause many errors. Hence, we strongly recommend people fine-tuning the model first.
We could have pushed the inference snippet for any model fine-tuned on top of smolvla_base however, but that is not really applicable here I think. wdyt?
There was a problem hiding this comment.
(originally posted in DMs on slack)
However, unfortunately we cannot use smolvla_base for inference directly, the only use case for smolvla_base for now is fine-tuning.
Interesting! What I would suggest:
- if
modelId is smolvla_base=> then provide the finetune command - if
modelId != smolvla_base=> then provide the "how to use" command + the finetune command
Since export const smolvla = (model: ModelData): string[] => [ is a method and you have access to ModelData you can easily add some simple logic like this to the snippet generation
I see there are 380+ models on https://huggingface.co/models?other=lerobot and only 1 is not finetuned (if I understand correctly?) so it would be a shame not to show the inference command for all of them. What do you think?
Uh oh!
There was an error while loading. Please reload this page.