feature: add `PulpProject` class for Pulp-based package repository support by vamahaja · Pull Request #2163 · ceph/teuthology

vamahaja · 2026-03-26T13:33:55Z

This PR adds a PulpProject class that extends GitbuilderProject so teuthology can use Pulp-hosted package repositories. It introduces new config use_artifacts, set it to pulp (globally in site config or per job) to use Pulp; other values keep the current Gitbuilder and Shaman behavior.

Configuration

use_artifacts: 'pulp' - selects the Pulp backend (alongside existing 'shaman' and the default Gitbuilder path when unset / other values).
pulp_host — Pulp server hostname (documented in siteconfig.rst).

Key changes

Implement PulpProject inheriting from GitbuilderProject.
Override _get_base_url() to use Pulp’s repository API.
Override _get_distro() for Pulp’s distro format.
Override _get_package_version() and _get_package_sha1() to read metadata from Pulp responses.
Update get_builder_project() so config.use_artifacts == 'pulp' returns PulpProject.
Update siteconfig (and related docs) for use_artifacts: 'pulp' and pulp_host.
Add mock test TestPulpProject to teuthology/test/test_packaging.py.

kshtsk · 2026-03-27T19:50:34Z

This looks good, do we have an instance running with pulp to check this PR?

vamahaja · 2026-03-30T04:30:39Z

This looks good, do we have an instance running with pulp to check this PR?

@kshtsk for review, I am working on deploying Pulp server on OCP cluster.

kshtsk · 2026-04-07T09:51:15Z

teuthology/openstack/setup-openstack.sh

    cat > ~/.teuthology.yaml <<EOF
 $archive_upload
-use_shaman: false
+use_artifacts: 'shaman'


The option use_shaman: false means that no shaman should be used. So if we switch to _use_artifacts, there should be None case to consider.

kshtsk · 2026-04-07T09:53:26Z

teuthology/config.py

        'baseurl_template': 'http://{host}/{proj}-{pkg_type}-{dist}-{arch}-{flavor}/{uri}',
-        'use_shaman': True,
-        'shaman_host': 'shaman.ceph.com',
+        'use_artifacts': 'shaman',


Maybe better to use artifacts_type instead?

Or even to match teuthology module name use: packaging_type?

kshtsk · 2026-04-07T10:07:37Z

teuthology/config.py

-        'use_shaman': True,
-        'shaman_host': 'shaman.ceph.com',
+        'use_artifacts': 'shaman',
+        'artifacts_host': 'shaman.ceph.com',


I suggest to start using endpoint URI instead of just hostname, so we have some kind of flexibility and avoid hardcoded paths, and for fast switch between artifacts type, maybe it is better to have api configuration separate:

shaman: endpoint: https://shaman.ceph.com/api pulp: endpoint: http://pulp.example.com/pulp/api/v3

What do you think? And you @zmc?

@zmc as discussed in weekly teuthology call, we can continue with below config -

artifacts: type: shaman | pulp endpoint: "https://<shaman|pulp>.ceph.com" username: <shaman|pulp>-username password: <shaman|pulp>-password

@kshtsk ^^^

I would prefer to stay with separate service tool based config when it comes to authentication, like it is done for fog and maas:

packaging_type: shaman | pulp | None shaman: endpoint: https://shaman.ceph.com/api username: password: pulp: endpoint: http://pulp.example.com/pulp/api/v3 username: password:

Why it is better?
Easy switch between each other. Probable benefit from selecting artifact type using overrides.

…pport Add a `PulpProject` subclass of `GitbuilderProject` so teuthology can use Pulp-hosted package repos when `config.use_artifacts` is `pulp`, alongside the existing `Gitbuilder` and `Shaman` artifact backends. Signed-off-by: Vaibhav Mahajan <vamahaja@redhat.com>

batrick

Looking good!!

batrick · 2026-04-09T17:06:36Z

teuthology/packaging.py

+        super(PulpProject, self).__init__(project, job_config, ctx, remote)
+
+        # Set the url for the pulp server.
+        self.pulp_server_url = f'http://{config.artifacts_host}'


Suggested change

self.pulp_server_url = f'http://{config.artifacts_host}'

self.pulp_server_url = f'https://{config.artifacts_host}'

?

If you need http:// for testing, use a config but default to https://.

batrick · 2026-04-09T17:09:07Z

teuthology/run.py

+    # If the job has a 'use_artifacts' key, use that value to override the global
+    # config's 'use_artifacts' value.
+    if config.get('use_artifacts') is not None:
+        teuth_config.use_artifacts = config['use_artifacts']


Suggested change

teuth_config.use_artifacts = config['use_artifacts']

teuth_config.use_artifacts = config['use_artifacts']

elif config.get('use_shaman') is not None:

# TODO: Add a logging warning about deprecation here

teuth_config.use_artifacts = 'shaman' if config['use_shaman'] else None

vamahaja requested a review from a team as a code owner March 26, 2026 13:33

vamahaja requested review from VallariAg and kamoltat and removed request for a team March 26, 2026 13:33

vamahaja added DNM feature labels Mar 26, 2026

vamahaja marked this pull request as draft March 26, 2026 13:34

vamahaja mentioned this pull request Mar 26, 2026

Alternate package providers #2135

Open

vamahaja linked an issue Mar 26, 2026 that may be closed by this pull request

Alternate package providers #2135

Open

deepssin requested review from amathuria, kshtsk and zmc March 27, 2026 04:15

vamahaja force-pushed the pulp-project branch 5 times, most recently from f60b0ef to b2e5898 Compare April 7, 2026 09:19

kshtsk reviewed Apr 7, 2026

View reviewed changes

vamahaja force-pushed the pulp-project branch from b2e5898 to 6a38b81 Compare April 7, 2026 11:33

vamahaja force-pushed the pulp-project branch from 6a38b81 to bb9ccff Compare April 7, 2026 11:58

batrick requested changes Apr 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: add `PulpProject` class for Pulp-based package repository support#2163

feature: add `PulpProject` class for Pulp-based package repository support#2163
vamahaja wants to merge 1 commit intoceph:mainfrom
vamahaja:pulp-project

vamahaja commented Mar 26, 2026 •

edited

Loading

Uh oh!

kshtsk commented Mar 27, 2026

Uh oh!

vamahaja commented Mar 30, 2026

Uh oh!

kshtsk Apr 7, 2026

Uh oh!

kshtsk Apr 7, 2026

Uh oh!

kshtsk Apr 7, 2026

Uh oh!

kshtsk Apr 7, 2026

Uh oh!

vamahaja Apr 9, 2026 •

edited

Loading

Uh oh!

kshtsk Apr 9, 2026

Uh oh!

batrick left a comment

Uh oh!

batrick Apr 9, 2026

Uh oh!

batrick Apr 9, 2026

Uh oh!

batrick Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	self.pulp_server_url = f'http://{config.artifacts_host}'
	self.pulp_server_url = f'https://{config.artifacts_host}'

-        teuth_config.use_artifacts = config['use_artifacts']
+        teuth_config.use_artifacts = config['use_artifacts']
+        elif config.get('use_shaman') is not None:
+             # TODO: Add a logging warning about deprecation here
+             teuth_config.use_artifacts = 'shaman' if config['use_shaman'] else None

Conversation

vamahaja commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kshtsk commented Mar 27, 2026

Uh oh!

vamahaja commented Mar 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vamahaja Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vamahaja commented Mar 26, 2026 •

edited

Loading

vamahaja Apr 9, 2026 •

edited

Loading