Skip to content

Device telemetry for benchmark #11301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Device telemetry for benchmark #11301

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 80 additions & 8 deletions extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,55 @@ phases:
echo "*.json tokenizer files found in /data/local/tmp/minibench/"
fi

- echo "Collect device state before running"
- echo "Mandatory Cool Down for 10 minutes"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell 'cat /sys/devices/system/cpu/cpu*/cpufreq/stats/time_in_state /sys/devices/system/cpu/cpu*/cpufreq/stats/trans_table' > $DEVICEFARM_LOG_DIR/state_before.txt
adb -s $DEVICEFARM_DEVICE_UDID shell 'sleep 600'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to schedule sleep on the device. Just make sure the job submissions is 10 minutes apar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or this is the only way to make sure really nothing else runs (like other users)


- echo "Collect Device Telemetry - CPU Scaling Configuration"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You want to collect these numbers before sleep not after

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nevermind. I see you are doing that after benchmark as well. Name the log file apprpriately. Like the pre benchmark should be prebenchmark suffixed.

- |
adb -s $DEVICEFARM_DEVICE_UDID shell '
for core_path in /sys/devices/system/cpu/cpu*/cpufreq; do
if [ -d "$core_path" ]; then
core_name=$(basename $(dirname $core_path))
governor=$(cat $core_path/scaling_governor 2>/dev/null || echo "N/A")
min_freq=$(cat $core_path/scaling_min_freq 2>/dev/null || echo "N/A")
max_freq=$(cat $core_path/scaling_max_freq 2>/dev/null || echo "N/A")
echo "$core_name | governor=$governor | min_freq=$min_freq | max_freq=$max_freq"
fi
done
' > $DEVICEFARM_LOG_DIR/telemetry_cpu_scaling_config.txt

- echo "Collect Device Telemetry - Thermal Status"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell dumpsys thermalservice > $DEVICEFARM_LOG_DIR/telemetry_thermal_status.txt

- echo "Collect Device Telemetry - CPU Frequency Statistics (pre-benchmark)"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell '
for core in /sys/devices/system/cpu/cpu*/cpufreq/stats; do
if [ -d "$core" ]; then
core_name=$(basename $(dirname $(dirname $core)))
echo "=== $core_name ==="
echo "--- time_in_state ---"
cat $core/time_in_state 2>/dev/null
echo "--- trans_table ---"
cat $core/trans_table 2>/dev/null
echo "--- total_trans ---"
cat $core/total_trans 2>/dev/null
echo ""
fi
done
' > $DEVICEFARM_LOG_DIR/telemetry_cpu_freq_stats_pre.txt

- echo "Collect Device Telemetry - Battery (pre-benchmark)"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell '
dumpsys battery
' > $DEVICEFARM_LOG_DIR/telemetry_battery_pre.txt

echo "Reset Battery Stats (pre-benchmark)"
adb -s $DEVICEFARM_DEVICE_UDID shell 'dumpsys batterystats --reset'
adb -s $DEVICEFARM_DEVICE_UDID shell 'dumpsys batterystats --enable full-wake-history'

- echo "Run benchmark"
- |
Expand All @@ -126,31 +172,57 @@ phases:

adb -s $DEVICEFARM_DEVICE_UDID shell dumpsys deviceidle force-idle
adb -s $DEVICEFARM_DEVICE_UDID shell dumpsys deviceidle unforce
adb -s $DEVICEFARM_DEVICE_UDID shell sleep 180

if [ -n "$BIN_FOUND" ]; then
adb -s $DEVICEFARM_DEVICE_UDID shell am start -W -n org.pytorch.minibench/.BenchmarkActivity \
--es "model_dir" "/data/local/tmp/minibench" \
--es "tokenizer_path" "/data/local/tmp/minibench/tokenizer.bin" \
--ei "num_iter" 5 --ei "num_warm_up_iter" 2
--ei "num_iter" 20 --ei "num_warm_up_iter" 8
elif [ -n "$MODEL_FOUND" ]; then
adb -s $DEVICEFARM_DEVICE_UDID shell am start -W -n org.pytorch.minibench/.BenchmarkActivity \
--es "model_dir" "/data/local/tmp/minibench" \
--es "tokenizer_path" "/data/local/tmp/minibench/tokenizer.model" \
--ei "num_iter" 5 --ei "num_warm_up_iter" 2
--ei "num_iter" 20 --ei "num_warm_up_iter" 8
elif [ -n "$JSON_FOUND" ]; then
adb -s $DEVICEFARM_DEVICE_UDID shell am start -W -n org.pytorch.minibench/.BenchmarkActivity \
--es "model_dir" "/data/local/tmp/minibench" \
--es "tokenizer_path" "/data/local/tmp/minibench/tokenizer.json" \
--ei "num_iter" 5 --ei "num_warm_up_iter" 2
--ei "num_iter" 20 --ei "num_warm_up_iter" 8
else
adb -s $DEVICEFARM_DEVICE_UDID shell am start -W -n org.pytorch.minibench/.BenchmarkActivity \
--es "model_dir" "/data/local/tmp/minibench"
fi

- echo "Collect device state after running"
- echo "Collect Device Telemetry - CPU Frequency Statistics (post-benchmark)"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell '
for core in /sys/devices/system/cpu/cpu*/cpufreq/stats; do
if [ -d "$core" ]; then
core_name=$(basename $(dirname $(dirname $core)))
echo "=== $core_name ==="
echo "--- time_in_state ---"
cat $core/time_in_state 2>/dev/null
echo "--- trans_table ---"
cat $core/trans_table 2>/dev/null
echo "--- total_trans ---"
cat $core/total_trans 2>/dev/null
echo ""
fi
done
' > $DEVICEFARM_LOG_DIR/telemetry_cpu_freq_stats_post.txt

- echo "Collect Device Telemetry - Battery (post-benchmark)"
- |
adb -s $DEVICEFARM_DEVICE_UDID shell 'cat /sys/devices/system/cpu/cpu*/cpufreq/stats/time_in_state /sys/devices/system/cpu/cpu*/cpufreq/stats/trans_table' > $DEVICEFARM_LOG_DIR/state_after.txt
adb -s $DEVICEFARM_DEVICE_UDID shell '
echo "=== POST-BENCHMARK BATTERY INFO ==="
dumpsys battery
echo ""
echo "=== APP BATTERY STATS (org.pytorch.minibench) ==="
dumpsys batterystats --charged org.pytorch.minibench
echo ""
echo "=== POWER CONSUMPTION SUMMARY ==="
dumpsys batterystats --charged | grep -E "(org.pytorch.minibench|Estimated power use|Time on battery|Total running time)"
' > $DEVICEFARM_LOG_DIR/telemetry_battery_post.txt

post_test:
commands:
Expand Down
Loading