WARN on and ignore Pulp-hosted shaman results when package_source: shaman AND sha1s of package repos lacking a container image

[djgalloway]

Problem

Security builds from the cve-pipeline are pushed to the lab-internal Pulp but still registered in shaman, with url/chacra_url pointing at Pulp. Shaman lists them first (newest), so for a sha1 that has both a cve build and a regular chacra build, ShamanProject picked the Pulp-hosted entry and 404'd fetching <chacra_url>/repo from the Pulp API:

teuthology-suite -s smoke -m smithi -c tentacle --newest 20 ...
...
INFO:teuthology.task.install.rpm:Pulling from https://pulp.front.sepia.ceph.com/pulp/content/repos/ceph/wip-yuriw-cve-dryrun_tentacle/66f67bca.../rocky/10/flavors/default/
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pulp.front.sepia.ceph.com/pulp/api/v3/repositories/rpm/rpm/019fedb6-4793-72bd-895b-d0aa8effb7ad/repo

Because teuthology-suite's schedule-time package check uses the same ShamanProject, --newest happily scheduled that sha1 too.

Changes

packaging: ignore Pulp-hosted shaman results

• ShamanProject filters out search results whose url/chacra_url are Pulp-hosted and uses the first usable (chacra) one. For the sha1 above, repo_url now resolves to https://3.chacra.ceph.com/repos/ceph/tentacle/66f67bca.../rocky/10/flavors/default/repo (verified against live shaman).
• With package_source: shaman, a sha1 that only has a Pulp build is treated as not found; new pulp_only property tells the two cases apart.
• package_source: pulp is unaffected: PulpProject never consults shaman, it searches the Pulp API directly.

suite: warn and back off when --newest hits a Pulp-only sha1

• --newest: a Pulp-only sha1 is skipped with a warning saying so and, if the run doesn't select package_source: pulp and defaults.cephadm.containers.image: quay-int.front.sepia.ceph.com/ceph-ci/ceph, how to opt in. Non-newest runs get the same hint in the error before failing on missing packages.
• teuthology-suite now reads the extra <config_yaml> files it's given (previously only forwarded to teuthology-schedule) so it can tell what the run selects.
• A job-level package_source is honored for the schedule-time checks (mirroring teuthology.run), so passing pulp.yaml makes --newest verify against Pulp instead of shaman. Falls back to the site package_source with a warning if no Pulp credentials are configured on the scheduling host.
• Docs / --newest help text updated.

suite: back off with --newest when a sha1 has packages but no container

• Release builds (ceph-release-pipeline → ceph-dev-pipeline with CI_CONTAINER=false) have packages and a completed shaman build record, but no ceph-ci container — e.g. tentacle 7f793731… right now: shaman says ready, quay.ceph.io/ceph-ci/ceph:7f793731… is 404. Nothing in shaman records this, so --newest would pick it and cephadm jobs would fail to pull.

• For jobs whose tasks use cephadm, ask the registry of the configured cephadm image (defaults/overrides.cephadm.containers.image) whether <image>:<sha1> exists (registry v2 manifest HEAD; anonymous bearer token fetched when challenged). If not:

  ceph sha1 <sha1> has packages but no container at <image>:<sha1>. Pending release build, or failed container build of a passed package build? Moving on to the next sha1 of <ref>

  and with --newest the sha1 is treated like missing packages so we back off. Without --newest it's only a warning. If the registry can't be queried, scheduling proceeds as before. Non-cephadm jobs never trigger the lookup, so non-container suites can still test release sha1s.

• Verified live: quay.ceph.io/ceph-ci/ceph:7f793731… → missing, :66f67bca… → exists; Docker Hub token flow works (alpine:latest).

Reviewer notes

• The Pulp-only detection relies on the cve-pipeline continuing to register its builds in shaman with Pulp URLs in chacra_url/url. If that ever stops, Pulp-only sha1s just look "not found" (no false positives, but the explanatory warning goes away). Noted in ShamanProject.pulp_only and util.hash_only_in_pulp.
• The quay-int hostname / pulp source name are module constants in teuthology/suite/run.py (INTERNAL_*); happy to make them site-config knobs if preferred.

Testing

tests/test_packaging.py, tests/suite, tests/scripts/test_suite.py, tests/task/test_install.py: 321 passed. New tests cover the exact two-entry shaman response from the failing job, the Pulp-only case, config-based detection of pulp/quay-int selection, an end-to-end --newest run that skips a Pulp-only sha1 with one warning, the registry helper (200/404/401+token/error), and an end-to-end --newest run that skips a container-less sha1.

[dmick]

Why are pulp-only builds registered with shaman at all?

[djgalloway]

Why are pulp-only builds registered with shaman at all?

The pulp GUI is half-baked as you know. Devs would have no way of knowing the status of their branch being built. We could make shaman interface with pulp directly but, for now, we're tricking shaman by giving it pulp URLs.