support custom science images #1163

MattShirley · 2025-09-23T15:37:15Z

Backend work for supporting custom TopCP images from whitelisted Docker registry repositories: ssl-hep/ServiceX_frontend#658

Allowed images can be defined using the app.allowedDockerRegistries value, where each key should be a Docker registry and each value should be a list of repositories/images to validate against. Each string in the list will be tested to see if it is a valid prefix of the custom image. If it is, it will be allowed. The default value will allow any tags for the Docker Hub sslhep/ repository.

app:
  allowedImagePrefixes:
    - "sslhep/"
    - "docker.io/sslhep"

Sample files that work with a custom TopCP image but not with the default/base image:

# reco.yaml
CommonServices:
  runSystematics: False

PileupReweighting: {}

EventCleaning: {}

Trigger:
    triggerChainsPerYear:
        '2015':
            - 'HLT_2mu6_bBmumuxv2'
            - 'HLT_2mu4_bBmumuxv2'
        '2016':
            - 'HLT_2mu6_bBmumux_BsmumuPhi_delayed'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_delayed_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_mu10_mu6_bBmumux_BsmumuPhi_delayed'
            - 'HLT_2mu6_bBmumuxv2'
            - 'HLT_2mu6_bBmumuxv2_delayed'
        '2017':
            - 'HLT_mu6_mu4_bBmumux_BsmumuPhi_L1BPH-2M9-MU6MU4_BPH-0DR15-MU6MU4'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bJpsimumu_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu10_bBmumuxv2'
        '2018':
            - 'HLT_mu6_mu4_bBmumux_BsmumuPhi_L1BPH-2M9-MU6MU4_BPH-0DR15-MU6MU4'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bJpsimumu_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu10_bBmumuxv2'
    noFilter: True
    noGlobalTriggerEff: True

Muons:
  - containerName: AnaMuons
    WorkingPoint:
      - selectionName: loose
        quality: Loose
        isolation: NonIso
      - selectionName: medium
        quality: Medium
        isolation: NonIso

BPhyVertex:
  - vertexName: 'BPHY28BsKKMuMuCandidates'
    decorations: []


BPHY28:
  - vertexName: 'BPHY28BsKKMuMuCandidates'
    inputMuons: 'AnaMuons'
    decorateTrackParameters: True
    decoratePixelHits: True
    decorateDedx: True

# After configuring each container, many variables will be saved automatically.
Output:
  treeName: 'reco'
  vars: []
  metVars: []
  containers:
      # Format should follow: '<suffix>:<output container>'
#      vtx_: 'BPHY28BsKKMuMuCandidates'
      mu_: 'AnaMuons'
      '': 'EventInfo'
  commands: []

AddConfigBlocks:
  - modulePath: 'TopCPToolkit.BPHY28Config'
    functionName: 'BPHY28Config'
    algName: 'BPHY28'
    pos: 'Output'
  - modulePath: 'TopCPToolkit.BPhyVertexConfig'
    functionName: 'BPhyVertexConfig'
    algName: 'BPhyVertex'
    pos: 'Output'

# spec.yaml
General:
  OutFilesetName: "fileset_ntuple_v1_af"
  OutputDirectory: "."

Sample:
  ## data18
  - Name: data18_periodD
    Dataset: !Rucio data18_13TeV:DAOD_BPHY28.44180527._000001.pool.root.1
    Query: !TopCP |
      reco=reco.yaml,
      image=sslhep/servicex_science_image_topcp:25.2.50-v2.18.0_v0.2

for more information, see https://pre-commit.ci

This reverts commit 86d536d.

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

…62-support-custom-topcp-images

for more information, see https://pre-commit.ci

…62-support-custom-topcp-images

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

codecov · 2025-10-28T20:31:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.56%. Comparing base (5905e54) to head (53b3af0).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1163      +/-   ##
===========================================
+ Coverage    86.25%   86.56%   +0.30%     
===========================================
  Files           95       95              
  Lines         3281     3304      +23     
  Branches       376      381       +5     
===========================================
+ Hits          2830     2860      +30     
+ Misses         377      373       -4     
+ Partials        74       71       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…62-support-custom-topcp-images

for more information, see https://pre-commit.ci

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

for more information, see https://pre-commit.ci

MattShirley · 2026-01-07T21:09:57Z

@ponyisi this PR has been updated to support CERN's GitLab instance. It also now validates all science images.

BenGalewsky · 2026-01-08T18:41:14Z

servicex_app/servicex_app/docker_repo_adapter.py

+        if registry == "docker.io":
+            return self._dockerhub_get_image_by_tag(repo, image, tag)
+
+        if registry == "gitlab-registry.cern.ch":


Is there really no way to make this generic? This is pretty 🤢

I added crane to the image and simplified this considerably

…crane

for more information, see https://pre-commit.ci

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

for more information, see https://pre-commit.ci

ponyisi · 2026-01-13T21:17:26Z

helm/servicex/values.yaml

  sqlalchemyEngineOptions: null
+  allowedDockerRegistries:
+    "docker.io":
+      allowedImagePrefixes:


is a prefix mandatory or can this just be "" ?

Right now a prefix is mandatory however it can just be "". I don't think we should include anything with "" in the default values.yaml but a cluster operator could choose to do so. Do you think we should keep KyungEon's GitLab repository by default? @kyungeonchoi do you know how easy it would be to setup a shared namespace in CERN's GitLab like sslhep-customimages that isn't tied to a specific user name?

Personally I'd tend to say we just whitelist the full gitlab-registry.cern.ch by default

servicex_app/servicex_app/docker_repo_adapter.py

ponyisi · 2026-01-13T21:28:56Z

servicex_app/servicex_app/resources/transformation/submit.py


-            request_rec.image = codegen_transformer_image
+            try:
+                jquery = json.loads(args["selection"])


This is not generically going to work for all codegens, many do not use JSON for their selection and even when they are they do not all have an "image" key.

This has been moved to the TopCP Code Gen's standard image setting logic.

ponyisi · 2026-01-13T21:31:23Z

servicex_app/servicex_app/resources/transformation/submit.py

+            registry = jquery.get("registry", "docker.io")
+
+            transformer_image = codegen_transformer_image
+            if "image" in jquery:


Again I don't really understand the point of doing this. If the codegen is doing the correct thing, codegen_transformer_image, which is under the control of the codegen which is interpreting the selection request, will have the image we want. We should not have any logic here that tries to peek into the selection, this must be kept opaque at this level. Basically, we should take codegen_transformer_image and prefix it with docker.io if necessary, that's it.

This has been moved to the TopCP Code Gen's standard image setting logic. Now the code moves immediately to validate the image name.

code_generator_TopCPToolkit/servicex/TopCP_code_generator/request_translator.py

…62-support-custom-topcp-images

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

for more information, see https://pre-commit.ci

code_generator_TopCPToolkit/servicex/TopCP_code_generator/request_translator.py

ponyisi · 2026-01-15T14:57:07Z

helm/servicex/values.yaml

  sqlalchemyEngineOptions: null
+  allowedDockerRegistries:
+    "docker.io":
+      allowedImagePrefixes:


Personally I'd tend to say we just whitelist the full gitlab-registry.cern.ch by default

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

for more information, see https://pre-commit.ci

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

ponyisi

OK, let's go

support custom TopCP images

34127c1

MattShirley linked an issue Sep 23, 2025 that may be closed by this pull request

Support custom TopCP images #1162

Closed

pre-commit-ci bot and others added 17 commits September 23, 2025 15:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

ba71014

for more information, see https://pre-commit.ci

resolve flake8 issues

86d536d

Revert "resolve flake8 issues"

b518d40

This reverts commit 86d536d.

Merge branch '1162-support-custom-topcp-images' of https://github.com…

c185637

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

fix broken test

8286d95

add back allowed_repos logic

2e706b7

remove accidental values changes

fe976f3

moving towards MVP (app server is failing due to bad mount)

e129256

update directory structure

0386476

tighten flake8

d0776b0

Merge branch 'develop' of https://github.com/ssl-hep/ServiceX into 11…

1cade42

…62-support-custom-topcp-images

remove comments and unused env vars

f1b7b43

[pre-commit.ci] auto fixes from pre-commit.com hooks

8a4fbb7

for more information, see https://pre-commit.ci

rewrite custom_docker_image logic

0fd050d

Merge branch 'develop' of https://github.com/ssl-hep/ServiceX into 11…

2d3bf8f

…62-support-custom-topcp-images

Merge branch '1162-support-custom-topcp-images' of https://github.com…

8fc3645

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

add back runTop_el

d7bb6cd

MattShirley and others added 10 commits October 28, 2025 15:33

remove old test files

e5024d0

fix local volume mount

0125f41

change reload to mountLocal

b49b9ee

Merge branch 'develop' of https://github.com/ssl-hep/ServiceX into 11…

f799896

…62-support-custom-topcp-images

update helm charts and other config stuff

6686336

[pre-commit.ci] auto fixes from pre-commit.com hooks

0f4f3f4

for more information, see https://pre-commit.ci

resolve flake8

8aeff9e

Merge branch '1162-support-custom-topcp-images' of https://github.com…

c9aaf95

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

[pre-commit.ci] auto fixes from pre-commit.com hooks

981504b

for more information, see https://pre-commit.ci

move boot.sh

28523ed

MattShirley changed the title ~~support custom TopCP images~~ support custom science images Jan 6, 2026

update documentation

7785682

BenGalewsky reviewed Jan 8, 2026

View reviewed changes

MattShirley and others added 6 commits January 9, 2026 15:39

resolve merge conflict

3d1f92f

replace bespoke registry APIs to validate docker images, instead use …

8b27d94

…crane

[pre-commit.ci] auto fixes from pre-commit.com hooks

e01ff98

for more information, see https://pre-commit.ci

mock crane subprocess call so tests pass without network access

6fd1bd0

Merge branch '1162-support-custom-topcp-images' of https://github.com…

eebcf92

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

[pre-commit.ci] auto fixes from pre-commit.com hooks

86cc261

for more information, see https://pre-commit.ci

ponyisi reviewed Jan 13, 2026

View reviewed changes

code_generator_TopCPToolkit/servicex/TopCP_code_generator/request_translator.py Show resolved Hide resolved

MattShirley and others added 7 commits January 13, 2026 18:57

Merge branch 'develop' of https://github.com/ssl-hep/ServiceX into 11…

875f7e3

…62-support-custom-topcp-images

separate custom image logic better between submit.py and TopCP code gen

661e4a0

Merge branch '1162-support-custom-topcp-images' of https://github.com…

4848d90

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

[pre-commit.ci] auto fixes from pre-commit.com hooks

d6e0121

for more information, see https://pre-commit.ci

add cache to and updates tests

888ead7

improve code coverage with new tests

1c440ba

[pre-commit.ci] auto fixes from pre-commit.com hooks

c5bf395

for more information, see https://pre-commit.ci

ponyisi reviewed Jan 15, 2026

View reviewed changes

MattShirley and others added 6 commits January 15, 2026 15:32

remove registry in favor of just image keyword

760a6f7

Merge branch '1162-support-custom-topcp-images' of https://github.com…

78d7337

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

[pre-commit.ci] auto fixes from pre-commit.com hooks

03367c8

for more information, see https://pre-commit.ci

remove final registry references

8053d3b

Merge branch '1162-support-custom-topcp-images' of https://github.com…

0b80225

…/ssl-hep/ServiceX into 1162-support-custom-topcp-images

fix test_src.py broken test

53b3af0

ponyisi approved these changes Jan 16, 2026

View reviewed changes

ponyisi merged commit 986a0da into develop Jan 16, 2026
137 of 152 checks passed

ponyisi deleted the 1162-support-custom-topcp-images branch January 16, 2026 19:33

ponyisi mentioned this pull request Jan 22, 2026

Check and return errors while validating science image. #555

Closed

Uh oh!

support custom science images #1163

support custom science images #1163

Uh oh!

Conversation

MattShirley commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MattShirley commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ponyisi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MattShirley commented Sep 23, 2025 •

edited

Loading

codecov bot commented Oct 28, 2025 •

edited

Loading