Skip to content

Benchmark Phi4-mini via optimum #11991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Benchmark Phi4-mini via optimum #11991

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/android-perf-private-device-experiment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,6 @@ jobs:
id-token: write
contents: read
with:
models: ${{ inputs.models || github.event_name == 'schedule' && 'Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
models: ${{ inputs.models || github.event_name == 'schedule' && 'Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,microsoft/Phi-4-mini-instruct,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'microsoft/Phi-4-mini-instruct' }}
devices: samsung_galaxy_s22_private
benchmark_configs: ${{ inputs.benchmark_configs }}
4 changes: 2 additions & 2 deletions .github/workflows/android-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ jobs:
# Separate default values from the workflow dispatch. To ensure defaults are accessible
# during scheduled runs and to provide flexibility for different defaults between
# on-demand and periodic benchmarking.
CRON_DEFAULT_MODELS: ${{ github.event_name == 'schedule' && 'mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l,meta-llama/Llama-3.2-1B,meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8,Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
CRON_DEFAULT_MODELS: ${{ github.event_name == 'schedule' && 'mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l,meta-llama/Llama-3.2-1B,meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8,Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,microsoft/Phi-4-mini-instruct,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
CRON_DEFAULT_DEVICES: samsung_galaxy_s22
run: |
set -eux
Expand Down Expand Up @@ -344,7 +344,7 @@ jobs:
git clone https://github.com/huggingface/optimum-executorch
pushd optimum-executorch
# There is no release yet, for CI stability, always test from the same commit on main
git checkout 4c3b18f6cca68c5ccff809131d570062723d7188
git checkout a3942627f5ac048e06b4b1d703b0a6a53bf6da5b
python install_dev.py --skip_override_torch
pip list

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/apple-perf-private-device-experiment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,6 @@ jobs:
id-token: write
contents: read
with:
models: ${{ inputs.models || github.event_name == 'schedule' && 'Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
models: ${{ inputs.models || github.event_name == 'schedule' && 'Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,microsoft/Phi-4-mini-instruct,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'microsoft/Phi-4-mini-instruct' }}
devices: apple_iphone_15_private
benchmark_configs: ${{ inputs.benchmark_configs }}
4 changes: 2 additions & 2 deletions .github/workflows/apple-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ jobs:
# Separate default values from the workflow dispatch. To ensure defaults are accessible
# during scheduled runs and to provide flexibility for different defaults between
# on-demand and periodic benchmarking.
CRON_DEFAULT_MODELS: ${{ github.event_name == 'schedule' && 'mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l,meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8,Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
CRON_DEFAULT_MODELS: ${{ github.event_name == 'schedule' && 'mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l,meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8,Qwen/Qwen3-0.6B,HuggingFaceTB/SmolLM2-135M,microsoft/Phi-4-mini-instruct,meta-llama/Llama-3.2-1B,allenai/OLMo-1B-hf' || 'Qwen/Qwen3-0.6B' }}
CRON_DEFAULT_DEVICES: apple_iphone_15
run: |
set -eux
Expand Down Expand Up @@ -349,7 +349,7 @@ jobs:
git clone https://github.com/huggingface/optimum-executorch
pushd optimum-executorch
# There is no release yet, for CI stability, always test from the same commit on main
git checkout 4c3b18f6cca68c5ccff809131d570062723d7188
git checkout a3942627f5ac048e06b4b1d703b0a6a53bf6da5b
${CONDA_RUN} python install_dev.py --skip_override_torch
pip list

Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/trunk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,7 @@ jobs:
google/gemma-3-1b-it,
Qwen/Qwen3-0.6B,
HuggingFaceTB/SmolLM2-135M,
microsoft/Phi-4-mini-instruct,
meta-llama/Llama-3.2-1B,
allenai/OLMo-1B-hf,
]
Expand Down Expand Up @@ -597,7 +598,7 @@ jobs:
git clone https://github.com/huggingface/optimum-executorch
pushd optimum-executorch
# There is no release yet, for CI stability, always test from the same commit on main
git checkout 4c3b18f6cca68c5ccff809131d570062723d7188
git checkout a3942627f5ac048e06b4b1d703b0a6a53bf6da5b
python install_dev.py --skip_override_torch
popd
pip list
Expand Down
Loading