Skip to content

add smolvla to model-libraries.ts #1529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jun 25, 2025
Merged

add smolvla to model-libraries.ts #1529

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0167612
add smolvla to model-libraries.ts
danaaubakirova Jun 13, 2025
2343fee
remove quotes
danaaubakirova Jun 13, 2025
39d5f50
update model-libraries-snippets.ts adding smolvla
danaaubakirova Jun 14, 2025
bab9770
update model-libraries.ts
danaaubakirova Jun 14, 2025
85e2ce1
Update model-libraries-snippets.ts
danaaubakirova Jun 14, 2025
00b8578
Update model-libraries-snippets.ts
danaaubakirova Jun 14, 2025
ebe7910
Update model-libraries-snippets.ts
danaaubakirova Jun 14, 2025
0f9d592
Update model-libraries.ts
danaaubakirova Jun 14, 2025
75517d1
update model-libraries-snippets.ts
danaaubakirova Jun 14, 2025
8ded89d
fix format
Jun 16, 2025
923df2d
Update packages/tasks/src/model-libraries-snippets.ts
danaaubakirova Jun 23, 2025
a68bbf6
Update packages/tasks/src/model-libraries.ts
danaaubakirova Jun 23, 2025
7f370c3
Update packages/tasks/src/model-libraries-snippets.ts
danaaubakirova Jun 23, 2025
f78b651
Update packages/tasks/src/model-libraries.ts
danaaubakirova Jun 23, 2025
64a8e3b
fix
Jun 23, 2025
15dc85d
fix
Jun 24, 2025
39edaa6
Merge branch 'main' into patch-1
danaaubakirova Jun 24, 2025
5ed372b
reorder to alphabetical order
Jun 24, 2025
17773dc
Update packages/tasks/src/model-libraries-snippets.ts
danaaubakirova Jun 24, 2025
d88c4f1
Update packages/tasks/src/model-libraries.ts
danaaubakirova Jun 24, 2025
6c805cb
Update packages/tasks/src/model-libraries.ts
danaaubakirova Jun 24, 2025
400c41f
Update packages/tasks/src/model-libraries.ts
danaaubakirova Jun 24, 2025
3bf9671
add the code for models that are not smolvla_base
Jun 24, 2025
b6e6919
smol fix
Jun 24, 2025
9a42a0d
reorder lerobot snippets
Jun 24, 2025
301e374
fix lint issues
Jun 24, 2025
f817655
Fix lint
pcuenca Jun 24, 2025
a21ee89
Merge branch 'main' into patch-1
danaaubakirova Jun 25, 2025
d50d2ce
Update packages/tasks/src/model-libraries-snippets.ts
Wauplin Jun 25, 2025
64b4a7d
Merge branch 'main' into patch-1
Wauplin Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions packages/tasks/src/model-libraries-snippets.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -762,6 +762,47 @@ print(output)`);
return snippets;
};

export const lerobot = (model: ModelData): string[] => {
if (model.tags.includes("smolvla")) {
const smolvlaSnippets = [
// Installation snippet
`# Install LeRobot from source
git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e .[smolvla]`,
// Finetune snippet
`# Launch finetuning on your dataset
python lerobot/scripts/train.py \\
--policy.path=${model.id} \\
--dataset.repo_id=lerobot/svla_so101_pickplace \\
--batch_size=64 \\
--steps=20000 \\
--output_dir=outputs/train/my_smolvla \\
--job_name=my_smolvla_training \\
--policy.device=cuda \\
--wandb.enable=true`,
];
if (model.id !== "lerobot/smolvla_base") {
// Inference snippet (only if not base model)
smolvlaSnippets.push(
`# Run the policy using the record function
python -m lerobot.record \\
--robot.type=so101_follower \\
--robot.port=/dev/ttyACM0 \\ # <- Use your port
--robot.id=my_blue_follower_arm \\ # <- Use your robot id
--robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \\ # <- Use your cameras
--dataset.single_task="Grasp a lego block and put it in the bin." \\ # <- Use the same task description you used in your dataset recording
--dataset.repo_id=HF_USER/dataset_name \\ # <- This will be the dataset name on HF Hub
--dataset.episode_time_s=50 \\
--dataset.num_episodes=10 \\
--policy.path=${model.id}`
);
}
return smolvlaSnippets;
}
return [];
};

export const tf_keras = (model: ModelData): string[] => [
`# Note: 'keras<3.x' or 'tf_keras' must be installed (legacy)
# See https://github.com/keras-team/tf-keras for more details.
Expand Down Expand Up @@ -1735,4 +1776,6 @@ audio = model.autoencoder.decode(codes)[0].cpu()
torchaudio.save("sample.wav", audio, model.autoencoder.sampling_rate)
`,
];


Copy link
Member

@pcuenca pcuenca Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually demonstrate simple inference snippets. Does that make sense for smolvla, or not really?

Copy link
Contributor

@Wauplin Wauplin Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree with @pcuenca that an inference example feels more natural at this stage since if a user has pushed their model, it means it has already been trained (at least I expect).

FYI, export const smolvla = (model: ModelData): string[] => [ returns a list so it could return both a training and an inference examples.

Copy link
Contributor Author

@danaaubakirova danaaubakirova Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I see what you mean. However, unfortunately we cannot use smolvla_base for inference directly, the only use case for smolvla_base for now is fine-tuning.
As it is not meant to perform zero-shot and will cause many errors. Hence, we strongly recommend people fine-tuning the model first.

We could have pushed the inference snippet for any model fine-tuned on top of smolvla_base however, but that is not really applicable here I think. wdyt?

Copy link
Contributor

@Wauplin Wauplin Jun 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(originally posted in DMs on slack)

However, unfortunately we cannot use smolvla_base for inference directly, the only use case for smolvla_base for now is fine-tuning.
Interesting! What I would suggest:

  • if modelId is smolvla_base => then provide the finetune command
  • if modelId != smolvla_base => then provide the "how to use" command + the finetune command

Since export const smolvla = (model: ModelData): string[] => [ is a method and you have access to ModelData you can easily add some simple logic like this to the snippet generation

I see there are 380+ models on https://huggingface.co/models?other=lerobot and only 1 is not finetuned (if I understand correctly?) so it would be a shame not to show the inference command for all of them. What do you think?

//#endregion
8 changes: 8 additions & 0 deletions packages/tasks/src/model-libraries.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,14 @@ export const MODEL_LIBRARIES_UI_ELEMENTS = {
filter: false,
countDownloads: `path_extension:"litertlm"`,
},
lerobot: {
prettyLabel: "LeRobot",
repoName: "LeRobot",
repoUrl: "https://github.com/huggingface/lerobot",
docsUrl: "https://huggingface.co/docs/lerobot",
filter: false,
snippets: snippets.lerobot,
},
liveportrait: {
prettyLabel: "LivePortrait",
repoName: "LivePortrait",
Expand Down
Loading