[llm] Add sentencepiece tokenizer support to llm runner

[larryliu0820]

Fixes #11618

Add sentencepiece tokenizer support

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11645

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 9df5f1a with merge base a1dec07 ():

NEW FAILURE - The following job has failed:

• Apple / build-benchmark-app / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[jackzhxng]

I think @kimishpatel said he had to make some additional changes on top of this to get it to work with Gemma3

[larryliu0820]

I think @kimishpatel said he had to make some additional changes on top of this to get it to work with Gemma3

Yep sync'ed with him will update the PR

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D76789606

[guangy10]

Will it be added automatically to the apps, e.g. benchmark apps? Particular for iOS app, do we need to add the source to the project tree?

[larryliu0820]

Will it be added automatically to the apps, e.g. benchmark apps? Particular for iOS app, do we need to add the source to the project tree?

Not sure, will check CI

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D76789606

[guangy10]

Will it be added automatically to the apps, e.g. benchmark apps? Particular for iOS app, do we need to add the source to the project tree?

Not sure, will check CI

Gemma-3 is disabled from running on the app. To check on CI, you can add gemma-3 back here:

executorch/.github/workflows/apple-perf-private-device-experiment.yml

Line 60 in 7503bb3

models: ${{ inputs.models || github.event_name == 'schedule' && 'Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}

, then schedule an ondemand run on your PR

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D76789606

[kimishpatel]

do validate against hf tokenizer + model's output

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D76789606