Device telemetry for benchmark

[guangy10]

This PR is establishing basic device telemetry for Android devices in the Benchmark Infra

Fill the DevX gap that we have been discussed here:#10983

The goal of establishing device telemetry are to:

1. Gather key stats for device health monitoring, which will be used to debug and root cause outliers. For example, perf drop due to thermal throttling, cpu scaling, etc.
2. Facilitate inventory management, as we will like be responsible managing devices in the private pool, the ability of knowing when to replace a problematic device is crucial

The device telemetry to be collected via this PR:

1. CPU scaling configuration

Whether the CPU scaling is locked or not on the device under test.
This is the one-time stats collected prior to the benchmark run, after mandatory cool down sleeping.

Here is the example of collected CPU scaling config for S22:

cpu0 | governor=walt | min_freq=614400 | max_freq=1363200
cpu1 | governor=walt | min_freq=614400 | max_freq=1363200
cpu2 | governor=walt | min_freq=614400 | max_freq=1363200
cpu3 | governor=walt | min_freq=614400 | max_freq=1363200
cpu4 | governor=walt | min_freq=633600 | max_freq=1996800
cpu5 | governor=walt | min_freq=633600 | max_freq=1996800
cpu6 | governor=walt | min_freq=633600 | max_freq=1996800
cpu7 | governor=walt | min_freq=806400 | max_freq=2284800

The governor walt is the dynamic sched common on Qcomm chip. min_freq != max_freq also shows the CPU scaling is not locked.

2. CPU frequency transitions

Record time_in_state, trans_table and total_trans before and after the benchmark run, then calculating the difference will show the frequency behavior specifically during your benchmark.
Important: The time_in_state, trans_table and total_trans data show cumulative statistics from system boot or last reset, so need to be reset prior to start benchmark jobs in order to get more accurate CPU frequency transitions during benchmark run. We need to collect them both pre-benchmark and post-benchmark.

Here is the example of collected CPU frequency transitions from S22:

=== cpu0 ===
--- time_in_state ---
307200 0
403200 0
518400 0
614400 72542
729600 1444
844800 639
960000 660
1075200 2960
1171200 328
1267200 202
1363200 8088
1478400 518
1574400 6
1689600 0
1785600 2057
--- trans_table ---
   From  :    To
         :    307200    403200    518400    614400    729600    844800    960000   1075200   1171200   1267200   1363200   1478400   1574400   1689600   1785600 
   307200:         0         0         0         0         0         0         0         0         0         0         0         0         0         0         0 
   403200:         0         0         0         0         0         0         0         0         0         0         0         0         0         0         0 
   518400:         0         0         0         0         0         0         0         0         0         0         0         0         0         0         0 
   614400:         0         0         0         0       746        50        24      1364        15         2       142        16         0         0         9 
   729600:         0         0         0       761         0        71        20       188         5         1        15         4         0         0         0 
   844800:         0         0         0       218        50         0        65       131        26         1         8         0         0         0         2 
   960000:         0         0         0       182        26        68         0        78        45        19        12         4         3         0         9 
  1075200:         0         0         0       849       176       216       224         0       156        70       286         6         2         1         3 
  1171200:         0         0         0        80        12        36        34       107         0        13        52         4         0         0         4 
  1267200:         0         0         0        39         5        14        20        24        32         0        52         3         0         0         3 
  1363200:         0         0         0       219        47        44        43        81        55        81         0         8         0         0         5 
  1478400:         0         0         0        13         3         1         6         8         6         2        12         0         0         0         1 
  1574400:         0         0         0         0         0         0         2         4         0         1         1         4         0         0         1 
  1689600:         0         0         0         0         0         0         0         0         0         0         0         0         3         0         0 
  1785600:         0         0         0         8         0         1         8         4         2         2         3         3         4         2         0 
--- total_trans ---
7593

=== cpu1 ===
...
  
--- total_trans ---
7995

3. The Thermal Stats

Record the thermal status of the device prior to the benchmark.
This is to ensure the device will not be over-heated prior to start benchmark. We may conditionally put the device for extra sleep or skip benchmarking if over-heating is detected. For the beginning, we will start with simplest approach by putting device to a mandatory sleep for 10 mins unconditionally.

Here is the example of collected thermal stats from S22:

IsStatusOverride: false
ThermalEventListeners:
	callbacks: 1
	killed: false
	broadcasts count: -1
ThermalStatusListeners:
	callbacks: 3
	killed: false
	broadcasts count: -1
Thermal Status: 0
Cached temperatures:
	Temperature{mValue=27.0, mType=5, mName=PA1THM, mStatus=0}
	Temperature{mValue=0.0, mType=2, mName=SUBBAT, mStatus=0}
	Temperature{mValue=0.0, mType=2, mName=SUBBATRAW, mStatus=0}
	Temperature{mValue=39.6, mType=0, mName=AP, mStatus=0}
	Temperature{mValue=36.3, mType=2, mName=BAT, mStatus=0}
	Temperature{mValue=36.9, mType=4, mName=USB, mStatus=0}
	Temperature{mValue=35.8, mType=3, mName=SKIN, mStatus=0}
	Temperature{mValue=40.8, mType=5, mName=PATHM, mStatus=0}
HAL Ready: true
HAL connection:
	ThermalHAL 2.0 connected: yes
Current temperatures from HAL:
	Temperature{mValue=39.6, mType=0, mName=AP, mStatus=0}
	Temperature{mValue=36.3, mType=2, mName=BAT, mStatus=0}
	Temperature{mValue=40.8, mType=5, mName=PATHM, mStatus=0}
	Temperature{mValue=35.8, mType=3, mName=SKIN, mStatus=0}
	Temperature{mValue=0.0, mType=2, mName=SUBBAT, mStatus=0}
	Temperature{mValue=0.0, mType=2, mName=SUBBATRAW, mStatus=0}
	Temperature{mValue=36.9, mType=4, mName=USB, mStatus=0}
Current cooling devices from HAL:
Temperature static thresholds from HAL:

4. Battery Status and Info

Record battery status before and after the benchmark.
The battery status and info will be used to determine the battery health, which typically useful to determine perf regression caused by lower battery level, battery mode, etc. This will report temperature as well, in tenths of degrees Celsius.
Besides determine device health, by comparing the battery level before and after the benchmark we can have a rough understand of the power consumption of running ML models on-device. Though today the primary perf metrics we are focusing are latency and accuracy, power consumption of a model is a crucial metric for on-device in that would affect users experience significantly. Collecting this metric will give us some early signal of about power efficiency.

Here is the example of collected battery stats from S22:

Current Battery Service state:
  AC powered: false
  USB powered: true
  Wireless powered: false
  Dock powered: false
  Max charging current: 0
  Max charging voltage: 0
  Charge counter: 4530000
  status: 5
  health: 2
  present: true
  level: 100
  scale: 100
  voltage: 4252
  temperature: 363
  technology: Li-ion
  batteryMiscEvent: 65536
  batteryCurrentEvent: 1048576
  mSecPlugTypeSummary: 2
  LED Charging: true
  LED Low Battery: true
  current now: -58
  charge counter: 4530000
  Adaptive Fast Charging Settings: false
  Super Fast Charging Settings: true
FEATURE_WIRELESS_FAST_CHARGER_CONTROL: true
  mWasUsedWirelessFastChargerPreviously: false
  mWirelessFastChargingSettingsEnable: true
LLB CAL: 20220520
LLB MAN: 20220520
LLB CURRENT: YEAR2025M6D3
LLB DIFF: 157
  mSavedBatteryBeginningDate: 0
SEC_FEATURE_BATTERY_FULL_CAPACITY: true
  mFullCapacityEnable: false
FEATURE_HICCUP_CONTROL: true
FEATURE_SUPPORTED_DAILY_BOARD: false
SEC_FEATURE_BATTERY_LIFE_EXTENDER: false
SEC_FEATURE_USE_WIRELESS_POWER_SHARING: true
 mProtectBatteryMode: 0
 mProtectionThreshold: 80
 mLtcHighThreshold: 95
 mLtcHighSocDuration: 10080
 mLtcReleaseThreshold: 75
[Not Battery Adaptive Protect Mode]
BatteryInfoBackUp
  mSavedBatteryAsoc: 93
  mSavedBatteryMaxTemp: 706
  mSavedBatteryMaxCurrent: 5792
  mSavedBatteryUsage: 89263
  mSavedFullStatusDuration: -1
  FEATURE_SAVE_BATTERY_CYCLE: true
  
=== APP BATTERY STATS (org.pytorch.minibench) ===
Estimated power use (mAh):
  Capacity: 4855, Rated: 4855, Typical: 5000, Computed drain: 0, actual drain: 0
  Global

Per process state tracking available: true
  Total cpu time reads: 1340
  Batching Duration (min): 14
    
On-battery energy consumer stats (microcoulombs) 
    Not supported on this device.

=== POWER CONSUMPTION SUMMARY ===
  Time on battery: 0ms (0.0%) realtime, 0ms (--%) uptime
  Time on battery screen off: 0ms (--%) realtime, 0ms (--%) uptime
  Time on battery screen doze: 0ms (--%)
  Estimated power use (mAh):      

The raw artifacts are downloadable via the CI, under DEVICEFARM_LOG_DIR:

├── Device_Files
└── Host_Machine_Files
    └── $DEVICEFARM_LOG_DIR
        ├── benchmark_results.json
        ├── instrument.log
        ├── result.etdump
        ├── telemetry_battery_post.txt
        ├── telemetry_battery_pre.txt
        ├── telemetry_cpu_freq_stats_post.txt
        ├── telemetry_cpu_freq_stats_pre.txt
        ├── telemetry_cpu_scaling_config.txt
        └── telemetry_thermal_status.txt

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11301

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a6b2e51 with merge base 8cfa858 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kimishpatel]

I dont follow the time_in_state part of the summary. That is supposed to be time in each frequency state, right? If so your output pasted in summary doesnt make sense

[kimishpatel]

We may conditionally put the device for extra sleep or skip benchmarking if over-heating is detected

Why do this conditionally? Why not just always have this

[kimishpatel]

We may conditionally put the device for extra sleep or skip benchmarking if over-heating is detected

Why do this conditionally? Why not just always have this

Ok I see that you are already doing this

[kimishpatel]

Why battery stats? I thought these devices are always plugged in and they should be. and if they are plugged in there isnt much we info we get out of battery measurements really

[kimishpatel]

Why do you need to schedule sleep on the device. Just make sure the job submissions is 10 minutes apar.

[digantdesai]

Or this is the only way to make sure really nothing else runs (like other users)

[kimishpatel]

You want to collect these numbers before sleep not after

[kimishpatel]

Oh nevermind. I see you are doing that after benchmark as well. Name the log file apprpriately. Like the pre benchmark should be prebenchmark suffixed.

[kimishpatel]

sleep doesnt need to be scheduled on device

[kimishpatel]

sleep doesnt need to be scheduled on device

but benchmark submission framework should make sure that after the benchmark is run, device is unavailable for certain amount of time. I dont know if just doing sleep will block the device. This has to be at benchmark infra level

[digantdesai]

Looks good at a high level. Logical next step is to write a python script and flag if something is off in these txt files and discard or flag the numbers if we see throttling or thermal interrupts or something along those lines. But I understand parsing, and processing these kernels logs can be pain, so I will leave it upto you how much to automate.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.