Plugin load fails depending on pythonpath name

[sethrj]

Check duplicate issues.

• Checked for duplicates

Goal

This seems to be similar to #1207 so I apologize if this is considered a duplicate.

I'm trying to test a new plugin (see celeritas-project/celeritas#1756) locally on my mac. I've got $DYLD_LIBRARY_PATH, $ROOT_LIBRARY_PATH, and $DD4HEP_LIBRARY_PATH set. Spack's environment helper also define a python venv for stricter environment isolation: its /opt/spack/opt/spack/tahoe/python-venv/1.0/ydmxvoj/bin/python3 is a symlink to the /opt/homebrew/opt/python@3.13/bin/python3.13 against which I built dd4hep.

When running dd4hep directly, the downstream plugin load fails:

Geant4UI         INFO  +++ ConstructSD> Install Geant4 control directory:/ddg4/ConstructSD/
Geant4Handle     ERROR Failed to create object of type DDcelerRunAction!
Traceback (most recent call last):
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/bin/ddsim", line 27, in <module>
    sys.exit(RUNNER.run())
             ~~~~~~~~~~^^
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDSim/DD4hepSimulation.py", line 376, in run
    action = DDG4_Action(kernel, action_dict["name"])
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDG4.py", line 213, in RunAction
    return Interface.createRunAction(kernel, str(nam), shared)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
cppyy.gbl.std.runtime_error: static dd4hep::sim::RunActionHandle dd4hep::sim::Geant4ActionCreation::createRunAction(dd4hep::sim::KernelHandle& kernel, const string& name_type, bool shared) =>
    runtime_error: Geant4Handle: Failed to create object of type DDcelerRunAction!

However, when I change the executable to /opt/homebrew/opt/python@3.13/bin/python3.13 $(which ddsim), the code works. What is going on? Is something in ROOT or DD4HEP looking at the equivalent of $(dirname $0)/../lib? Because the executables are exactly the same, just linked to different paths. And why does the "external" plugin load fail rather than any of the dd4hep or ROOT plugins that are clearly loaded? (@pcanal)

Thanks!

Operating System and Version

macos 26.2

compiler

apple clang 17

ROOT Version

6.38

DD4hep Version

1.33

Reproducer

Spack install dd4hep, try to run a dd4hep script that uses an external plugin

Additional context

zb5na2e dd4hep@1.33+ddalign~ddcad+ddcond+dddetectors+dddigi~ddeve+ddg4+ddrec~debug~doc+edm4hep~geant4units+hepmc3~hepmc3-gz~ipo~lcio~tbb~utilityapps~xercesc build_system=cmake build_type=Release cxxstd=20 generator=ninja
sdtgn2a     apple-clang@17.0.0 build_system=bundle
anx4mzx     boost@1.89.0~atomic~charconv~chrono~clanglibcpp~cobalt~container~context~contract~conversion~coroutine~date_time~debug~exception~fiber~filesystem~graph~graph_parallel~icu+iostreams~json~locale~log~math~mpi~mqtt5+multithreaded~nowide~numpy~pic~program_options~python~random~regex~serialization+shared~signals2~singlethreaded~stacktrace~system~taggedlayout~test~thread~timer~type_erasure~url~versionedlayout~wave build_system=generic cxxstd=20 patches:=a440f96 visibility=hidden
nooaoel         bzip2@1.0.8~debug~pic+shared build_system=generic
qfwo72z         xz@5.8.1~pic build_system=autotools libs:=shared,static
bhoi2cp         zlib-ng@2.2.4+compat~ipo+new_strategies+opt+pic+shared build_system=cmake build_type=Release generator=ninja
pxmgqr7         zstd@1.5.7~ipo~programs build_system=cmake build_type=Release generator=ninja libs:=shared,static
wsrxx6f     cmake@4.1.1~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release
e6qqck2     compiler-wrapper@1.0 build_system=generic
xvadrab     edm4hep@0.99.4~ipo+json build_system=cmake build_type=Release cxxstd=20 generator=ninja
7mlcviu         nlohmann-json@3.12.0~ipo+multiple_headers build_system=cmake build_type=Release generator=make
nvnw66o         py-jinja2@3.1.6~i18n build_system=python_pip
sss75pn             py-flit-core@3.12.0 build_system=python_pip
4jlbtiw             py-markupsafe@3.0.2 build_system=python_pip
gkt5kfa         py-pyyaml@6.0.3+libyaml build_system=python_pip
kw6jrf2             libyaml@0.2.5 build_system=autotools
djjnd7e             py-cython@3.2.3 build_system=python_pip
gykadvh     geant4@11.3.2+data~hdf5~ipo~motif~opengl~qt~tbb+threads~vecgeom~vtk~x11 build_system=cmake build_type=Release cxxstd=20 generator=ninja
aiprex5         clhep@2.4.7.2~cms~ipo build_system=cmake build_type=Release cxxstd=20 generator=make
lzxv45i         expat@2.7.3~ipo~libbsd+shared build_system=cmake build_type=Release generator=ninja
vpwbrv6         geant4-data@11.3.0+nudexlib+tendl+urrpt build_system=bundle
nsakeyw             g4abla@3.3 build_system=generic
pfuelda             g4channeling@1.0 build_system=generic
xxssutw             g4emlow@8.6.1 build_system=generic
v4yq6hy             g4ensdfstate@3.0 build_system=generic
cjjrty2             g4incl@1.2 build_system=generic
qplsyw3             g4ndl@4.7.1 build_system=generic
fw6ji6p             g4nudexlib@1.0 build_system=generic
7hojf3o             g4particlexs@4.1 build_system=generic
wxyftmj             g4photonevaporation@6.1 build_system=generic
db55rsz             g4pii@1.3 build_system=generic
chvmahw             g4radioactivedecay@6.1.2 build_system=generic
tpntlro             g4realsurface@2.2 build_system=generic
sxxwqoi             g4saiddata@2.0 build_system=generic
aw2yc7h             g4tendl@1.4 build_system=generic
bt3uxg3             g4urrpt@1.1 build_system=generic
oz7lsli         xerces-c@3.3.0 build_system=autotools cxxstd=20 netaccessor=curl transcoder=macos
4vttdtt             curl@8.15.0~gssapi~ipo~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=cmake build_type=Release generator=ninja libs:=shared,static tls:=openssl
nkysssy                 nghttp2@1.67.1 build_system=autotools
dqyvvfs                     diffutils@3.12 build_system=autotools
fbe3hi3                         libiconv@1.18 build_system=autotools libs:=shared,static
wek257z                 openssl@3.5.2~docs+shared build_system=generic certs=mozilla
zdhucai                 perl@5.40.2~cpanm+opcode+open+shared+threads build_system=generic
522lsrq             gnuconfig@2024-07-27 build_system=generic
inzyjpe     hepmc3@3.3.1~interfaces~ipo~protobuf~python~rootio build_system=cmake build_type=Release generator=make
z2s764f         cmake@4.1.1~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release
2k2iwe6         gmake@3.81~guile build_system=generic
v7sb52q     ninja@1.13.1+re2c build_system=generic
cifseb7     podio@1.6~datasource~ipo~rntuple~sio build_system=cmake build_type=Release cxxstd=20 generator=ninja
w2bfm2a         fmt@12.1.0~ipo+pic~shared build_system=cmake build_type=Release cxxstd=20 generator=make
y6u6yap         py-graphviz@0.20.3~dev~docs build_system=python_pip
flpcwgy         py-tabulate@0.9.0 build_system=python_pip
smzjzmr     py-pytest@9.0.0 build_system=python_pip
dtdw67a         py-iniconfig@2.1.0 build_system=python_pip
mxjo7t7             py-hatch-vcs@0.5.0 build_system=python_pip
el3w2q6             py-hatchling@1.27.0 build_system=python_pip
zy3dnoc                 py-pathspec@0.12.1 build_system=python_pip
4w6jmgk                 py-trove-classifiers@2025.9.11.17 build_system=python_pip
7u6q2w7                     py-calver@2025.4.17 build_system=python_pip
hvpdbr4         py-packaging@25.0 build_system=python_pip
5n7ec5m         py-pip@25.1.1 build_system=generic
smjqg3q         py-pluggy@1.6.0 build_system=python_pip
d74ch63         py-pygments@2.19.2 build_system=python_pip
jwr43q3         py-setuptools@80.9.0 build_system=generic
xlcpplq         py-setuptools-scm@8.2.1+toml build_system=python_pip
rojrlzl             git@2.51.0+man+nls+perl+subtree~tcltk build_system=autotools
umk7dgy         py-wheel@0.45.1 build_system=generic
zu4diyj     python@3.13.7+bz2+ctypes+dbm~debug~freethreading+libxml2+lzma~optimizations+pic+pyexpat~pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic
ydmxvoj     python-venv@1.0 build_system=generic
63euguv     root@6.38.00~aqua~arrow~cuda~cudnn~cxxmodules~daos~davix~dcache~emacs~examples~fftw~fits~fortran+gdml+geom~geombuilder+gminimal~graphviz+gsl~http~ipo+math+minuit~mlp~opengl~pythia8+python~qt6~r+roofit+root7+rpath~shadow~spectrum~sqlite~ssl~tbb+tiff~tmva~tmva-cpu+tmva-cudnn~tmva-gpu~tmva-pymva~tmva-sofie+tpython+unuran~vc+vdt~veccore~webgui~x+xml~xrootd build_system=cmake build_type=Release cxxstd=20 generator=make patches:=22af347
bsi4th7         freetype@2.14.1~ipo+pic+shared build_system=cmake build_type=Release generator=ninja
xz4vqru         giflib@5.2.2 build_system=makefile patches:=5cc0447
h35penr         gsl@2.8~external-cblas+pic+shared build_system=autotools
sxzlkmx         libice@1.1.2 build_system=autotools
ev6t5e7             util-macros@1.20.1 build_system=autotools
l56vqlm             xproto@7.0.31 build_system=autotools
b243xvj             xtrans@1.6.0 build_system=autotools
677y4or         libjpeg-turbo@3.0.4~ipo~jpeg8~partial_decoder+pic build_system=cmake build_type=Release generator=make libs:=shared,static
wo4tj6r             nasm@2.16.03 build_system=autotools
vrqehaf         libpng@1.6.47~ipo~pic build_system=cmake build_type=Release generator=ninja libs:=shared,static
pqvlbco         libtiff@4.7.0+ccitt~ipo~jbig+jpeg~jpeg12~lerc~libdeflate+logluv~lzma+lzw+next~old-jpeg~opengl+packbits~pic~pixarlog+shared+thunder~webp+zlib~zstd build_system=cmake build_type=Release generator=ninja
w4tzpet         libxml2@2.13.5~http~ipo+pic~python+shared build_system=cmake build_type=Release generator=ninja
ixs2vus         lz4@1.10.0~ipo+pic build_system=cmake build_type=Release generator=ninja libs:=shared,static
4od2mo5         ncurses@6.0.20150808~symlinks+termlib abi:=5 build_system=autotools
5nkqijm         pcre@8.45~jit+multibyte+pic+shared+static+utf build_system=autotools
oio7pqf         pkgconf@2.5.1 build_system=autotools
f755nbg         unuran@1.11.0~gsl+rngstreams+shared build_system=autotools
mas57d6             rngstreams@1.0.1 build_system=autotools
hcdoyqf         vdt@0.4.6~ipo~preload build_system=cmake build_type=Release generator=ninja
3chu5om         xxhash@0.8.3 build_system=makefile

[andresailer]

When running dd4hep directly, the downstream plugin load fails:

What do you mean with dd4hep directly? Do you mean ddsim? Does SIP purge the path because /usr/bin/python3 is involved (or something like this)?

PS:

DD4hep/DDG4/python/DDSim/bin/ddsim.in.py

Line 1 in de2efeb

#!/usr/bin/env @PYTHON_INTERPRETER_BINARY@

That /usr/bin/env would be enough to purge some env variables.

Are you sourcing your thisDDCeler.sh file (or what it might be called).

[sethrj]

Thanks @andresailer , I meant ddsim directly. I was executing launch-ddceler.sh as a script (not sourcing it). I might've been doing something else wrong with DD4HEP_LIBRARY_PATH vs DYLD_LIBRARY_PATH vs ...

I think an ideal solution would be like the ROOT_LIBRARY_PATH (root-project/root#7031): always use DD4HEP_LIBRARY_PATH (in addition to the current system-dependent dynamic linker paths) for looking up DD4hep plugins. That would simplify debugging and improve cross-platform portability.

[andresailer]

But is the problem that your library wasn't found, or that some library your library depends on wasn't found? I don't want to re-implement the dynamic linker...

[sethrj]

Trying to reproduce this again (there were too many variables: dd4hep path, celeritas plugin, dyld path), I think you're right about env purging $DYLD_LIBRARY_PATH (i.e., the error was ultimately the same as #1207):

$ echo $DYLD_LIBRARY_PATH
/Users/seth/Code/celeritas-temp/build/lib
$ echo $DD4HEP_LIBRARY_PATH
/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib
$ ddsim --compactFile="./input/Preshower.xml" --steering="./input/steeringFile.py" >/dev/null
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Info in <TGeoManager::TGeoManager>: Geometry default, Detector Geometry created
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::SetTopVolume>: Top volume is world_volume. Master volume is world_volume
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 1, max placements = 2
Info in <TGeoManager::CloseGeometry>: 3 nodes/ 4 volume UID's in Detector Geometry
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Traceback (most recent call last):
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/bin/ddsim", line 27, in <module>
    sys.exit(RUNNER.run())
             ~~~~~~~~~~^^
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDSim/DD4hepSimulation.py", line 376, in run
    action = DDG4_Action(kernel, action_dict["name"])
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDG4.py", line 213, in RunAction
    return Interface.createRunAction(kernel, str(nam), shared)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
cppyy.gbl.std.runtime_error: static dd4hep::sim::RunActionHandle dd4hep::sim::Geant4ActionCreation::createRunAction(dd4hep::sim::KernelHandle& kernel, const string& name_type, bool shared) =>
    runtime_error: Geant4Handle: Failed to create object of type CelerRun!
$ python $(which ddsim) --compactFile="./input/Preshower.xml" --steering="./input/steeringFile.py" >/dev/null
# success

I also encountered this along the way:

$ export DD4HEP_LIBRARY_PATH=/Users/seth/Code/celeritas-temp/build/lib:$DD4hep_DIR/lib
$ ddsim --compactFile="./input/Preshower.xml" --steering="./input/steeringFile.py" >/dev/null
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libCore.so for gErrorIgnoreLevel
Info in <TGeoManager::TGeoManager>: Geometry default, Detector Geometry created
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::SetTopVolume>: Top volume is world_volume. Master volume is world_volume
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 1, max placements = 2
Info in <TGeoManager::CloseGeometry>: 3 nodes/ 4 volume UID's in Detector Geometry
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
WARNING: cannot load libCelerRun.dylib for factory CelerRun
WARNING: dlopen(libCelerRun.dylib, 0x0009): tried: 'libCelerRun.dylib' (no such file)'/System/Volumes/Preboot/Cryptexes/OSlibCelerRun.dylib' (no such file), '/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/libCelerRun.dylib' (no such file) ..... '/usr/lib/libCelerRun.dylib' (no such file, not in dyld cache), 'libCelerRun.dylib' (no such file)
Traceback (most recent call last):
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/bin/ddsim", line 27, in <module>
    sys.exit(RUNNER.run())
             ~~~~~~~~~~^^
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDSim/DD4hepSimulation.py", line 376, in run
    action = DDG4_Action(kernel, action_dict["name"])
  File "/opt/spack/opt/spack/tahoe/dd4hep/1.33/zb5na2e/lib/python3.13/site-packages/DDG4.py", line 213, in RunAction
    return Interface.createRunAction(kernel, str(nam), shared)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
cppyy.gbl.std.runtime_error: static dd4hep::sim::RunActionHandle dd4hep::sim::Geant4ActionCreation::createRunAction(dd4hep::sim::KernelHandle& kernel, const string& name_type, bool shared) =>
    runtime_error: Geant4Handle: Failed to create object of type CelerRun!

so some part of the code was OK with using DD4HEP_LIBRARY_PATH but another part failed the lookup.

I think that a DD4HEP_LIBRARY_PATH or DD4HEP_PLUGIN_PATH variable, which could be simply implemented in the python launcher similar to https://github.com/AIDASoft/DD4hep/blob/master/DDG4/python/DDG4.py#L33-L36 , would be a straightforward and robust way to prevent these kind of errors.

[andresailer]

Can you just give me a full reproducer on macOS that doesn't use spack so I can see how to fix this?

[sethrj]

@andresailer I think the issue is independent of spack, and since the failure involves loading a downstream plugin I'm not sure how to give you a reproducer for that. I'll just close this issue since I can't reproduce the issue of the python/exec path affecting the ROOT/DD4HEP lookup path, though I suspect it has to do with how ROOT resolves lookup paths based on executable_path on macos (see https://github.com/root-project/root/blob/cb54a79c90606b2e09ce6c75bb95fd20430718aa/interpreter/cling/lib/Interpreter/DynamicLibraryManager.cpp#L87 , perhaps @pcanal has a thought).

[pcanal]

(see root-project/root@cb54a79/interpreter/cling/lib/Interpreter/DynamicLibraryManager.cpp#L87)

This is only one of the ways, I think. eg. see root-project/root@47caba5.

Although, I am not clearly seeing the link between the location of the executable (and the consequence on the initial set of path searched) and external plugin library that apriori should not be in a directory related to that executable.