Add modular hw-probe runner (Debian/Ubuntu + Docker) with install/update, offline save, upload, auto-extract, and network checks

[smuppand]

This PR introduces a fully-modular hw-probe runner under Runner/tools/hw-probe/ with shared utilities in Runner/utils/. It supports Debian/Ubuntu native and Docker execution, handles install/update (incl. specific versions), works offline (save only) or online (upload), and can auto-extract the generated report. All logging and basic checks reuse our existing functestlib.sh (no duplicated infra).

Why

• Standardize hardware probing across environments.
• One-line UX to install deps, run probes, save artifacts, and (optionally) upload.
• Make behavior reliable in CI (network checks, explicit output paths, extraction).

What’s included

New CLI: Runner/tools/hw-probe/run.sh (POSIX sh)

• --mode local|docker

• --upload yes|no (auto-disables when offline)

• --out DIR (default ./hw-probe_out)

• --extract yes|no (auto-unpacks hw.info.txz to OUT/extracted-)

• --install, --version , --update, --list-versions, --deps-only

• --probe-args ""

• --dry-run, --verbose

Shared libs (POSIX sh):

• utils/lib_common.sh – tiny helpers (root/sudo, ensure_dir, OS detect, network check via check_network_status)

• utils/lib_apt.sh – APT install/update/version helpers (guarded by network checks)

• utils/lib_docker.sh – docker presence/install/run; skip pulls when offline, require local image

• utils/lib_hwprobe.sh – install/update/run (local & docker), save with -save (correct flag), parse “Probe URL”, print saved path, auto-extract when requested

Docs: Runner/tools/hw-probe/README.md (usage, examples, artifacts, offline tips, Docker notes, troubleshooting)

Key details

Uses functestlib.sh for all logging (log_info, log_warn, log_error, log_fail, log_skip) and check_dependencies; prefers check_network_status when available.

No reimplementation of logging/deps logic; libs are minimal and shellcheck-friendly.

Corrects flag usage: we use hw-probe -save DIR (not -dump) so artifacts persist deterministically.

Local runs use sudo -E automatically when not root.

Docker runs mirror upstream’s guidance: --privileged --net=host --pid=host and read-only host mounts (/dev, /lib/modules, /etc/os-release, /var/log) plus -v OUT:/out to persist results.

If --upload yes but offline, runner downgrades to no for that run (logs a warning).

After each run, we print where the archive was saved and (if uploaded) the probe URL. With --extract yes, we unpack the .txz to OUT/extracted-.

Usage examples

Local, save only, then auto-extract

./run.sh --mode local --upload no --extract yes

Local, upload + extract

./run.sh --mode local --upload yes --extract yes

Docker, save only, custom out dir

./run.sh --mode docker --upload no --out ./out --extract yes

Ensure latest hw-probe (if missing) then run

./run.sh --install --mode local

Install a specific version

./run.sh --install --version 1.6

Update to latest

./run.sh --update

List available versions

./run.sh --list-versions

Test plan

• Debian/Ubuntu: local run with/without --upload

• Offline mode: install/update guarded; --upload auto-disabled

• Docker mode: image pull online; offline requires local image; artifacts saved under OUT

• Auto-extraction: creates OUT/extracted- and unpacks hw.info.txz

• ShellCheck: POSIX sh; dynamic-source warnings only (expected for sourcing)

• Reuse of existing infra: logging/dep checks via functestlib.sh

[lool]

meh :)

Would this make shellcheck happy enough?
(. /etc/os-release && echo "${ID_LIKE:-$ID}")

[smuppand]

Sourcing /etc/os-release in a subshell still trips SC1091 unless we run ShellCheck with -x or add a directive. Easiest way to be noise-free without directives is to parse the file instead of sourcing it.

[lool]

Looks like hdparm is a Depends of hw-probe already; I wonder how was this list generated? How would it be kept up-to-date?

[lool]

I guess hw-probe is being installed by this script; I wonder if the script should simply rely on the Recommends of that package to pull all the right utilities (albeit some don't seem to be listed like lsscsi)

[smuppand]

Agree. We can stop hardcoding a deps list and just install hw-probe (APT already pulls Depends and, by default on Debian/Ubuntu, Recommends). We’ll keep an opt-in knob for additional utilities if needed later, but we won’t install them by default.

[lool]

urgh, do we really need to run in environments that don't have gnu tar?

[smuppand]

Since we target Debian/Ubuntu, GNU tar with -J is available. We can simplify and remove the bsdtar/xz fallbacks. We already check for tar in run.sh when --extract yes.

[lool]

This looks clean, but seems super complicated – parsing the output or log to figure where this was saved; is there no way to tell hw-probe to save somewhere specific? Or could we send a patch to do so? Seems painful to extract the info like this

[smuppand]

We already tell it where to save: -save "$out". Upstream uses a deterministic filename hw.info.txz inside that directory. So we can use $out/hw.info.txz directly and keep the log parse only as a fallback (defense in depth).

[lool]

If we are using a docker image, shouldn't it contain all the probing tools we need already?

[smuppand]

Yes. For Docker mode we don’t install any host tools—only ensure Docker itself exists. The linuxhw/hw-probe image bundles what it needs, we pass mounts/privileged flags so it can see host devices. Our current flow respects that (no host deps installation for Docker runs).

We’ll clarify this in the README (“Docker mode does not install host utilities, image contains them”).

[lool]

Nice! So this is my first review here, I might have to ask some basic questions as I get familiar with the assumptions here

Here are my notes in no particular order:

1. the first line of the commit messages is a bit long; I guess it's just a draft and would be cleaned up, I see other commits even seem to use semantic git commits, nice!

2. a description of the "why" would be nice somewhere

3. the shell code is super clean, big kudos for the style!

4. I don't understand why we support installing the package or using the docker image; is it to compare? are there pros and cons to each?

5. I also don't understand why we support optional core utilities like tar and the like being absent; is this for portability to BSD or something? Seems a bit extreme to have to carry this deep in shell scripts

6. this is not really feedback on this specific test, but more on what I infer from the architecture: I like how the shell scripts are nicely independent in this repo, even if they use some common libraries; it looks like each script will pull its dependencies as it starts, which I find a bit weird. I would either expect:

• we build an image with everything in it, and then it gets used or tests are skipped
• or we embed the dependencies and don't touch the image
• or we dynamically install everything missing for what we want to run at the start, then run all tests; that way, we control internet accesses during the run, and any network / package install errors are flushed out early (downside: can be long to install all deps before seeing some early tests fail)

Perhaps this could be implemented with dependency injection: each test would provide a few functions – "can_run()", "list_missing_os_pkgs()", "run_tests()" etc. and the main testsuite would call the tests' functions to report their needs. Or the dependencies would be metadata next to the tests that get processed by the test itself or by the testsuite.

7. I find the optional upload feature nice, but I'm curious how we'd use it; would we setup separate jobs where it's enabled?

Again, apologies for my naive questions, this looks great!

[smuppand]

Nice! So this is my first review here, I might have to ask some basic questions as I get familiar with the assumptions here

Here are my notes in no particular order:

1. the first line of the commit messages is a bit long; I guess it's just a draft and would be cleaned up, I see other commits even seem to use semantic git commits, nice!
2. a description of the "why" would be nice somewhere
3. the shell code is super clean, big kudos for the style!
4. I don't understand why we support installing the package or using the docker image; is it to compare? are there pros and cons to each?

Native (APT):
Pros: integrates with host, smallest runtime overhead, works without Docker.
Cons: needs root & network (first install), relies on host packages.

Docker:
Pros: self-contained userspace (image has tools), reproducible, minimal host mutation.
Cons: needs Docker, --privileged + host mounts, initial image pull.

5. I also don't understand why we support optional core utilities like tar and the like being absent; is this for portability to BSD or something? Seems a bit extreme to have to carry this deep in shell scripts

Totally fair. We target Debian/Ubuntu, so GNU tar with -J is effectively guaranteed. I’ve simplified extraction accordingly:

_hwprobe_extract_txz() now requires GNU tar (tar -xJf).

We also require tar up front when --extract yes is used.

6. this is not really feedback on this specific test, but more on what I infer from the architecture: I like how the shell scripts are nicely independent in this repo, even if they use some common libraries; it looks like each script will pull its dependencies as it starts, which I find a bit weird. I would either expect:

• we build an image with everything in it, and then it gets used or tests are skipped

This is the good idea and we are currently following this apporach in current qcom-linux-testkit as well.

• or we embed the dependencies and don't touch the image
• or we dynamically install everything missing for what we want to run at the start, then run all tests; that way, we control internet accesses during the run, and any network / package install errors are flushed out early (downside: can be long to install all deps before seeing some early tests fail)

Perhaps this could be implemented with dependency injection: each test would provide a few functions – "can_run()", "list_missing_os_pkgs()", "run_tests()" etc. and the main testsuite would call the tests' functions to report their needs. Or the dependencies would be metadata next to the tests that get processed by the test itself or by the testsuite.

7. I find the optional upload feature nice, but I'm curious how we'd use it; would we setup separate jobs where it's enabled?

We’ll expose it in CI via a matrix or separate job:

UPLOAD=no by default (save artifacts locally for later inspection).

UPLOAD=yes in an opt-in job (e.g., nightly / lab runs) to seed linux-hardware.org the runner prints the URL for traceability.

This keeps most CI jobs fast and isolated while allowing a dedicated pipeline to contribute probes.

Again, apologies for my naive questions, this looks great!