Completely disable outside python

[Moelf]

 Package         Version  Build               Channel           Size
───────────────────────────────────────────────────────────────────────
  Reinstall:
───────────────────────────────────────────────────────────────────────

  o libstdcxx-ng   11.3.0  h239ccf8_18         conda-forge     Cached
  o python         3.10.6  h582c2e5_0_cpython  conda-forge     Cached

  Summary:

  Reinstall: 1 packages

  Total download: 0 B

───────────────────────────────────────────────────────────────────────


Transaction starting
Changing libstdcxx-ng-11.3.0-h239ccf8_18 ==> libstdcxx-ng-11.3.0-h239ccf8_18
Changing python-3.10.6-h582c2e5_0_cpython ==> python-3.10.6-h582c2e5_0_cpython
Transaction finished
Python path configuration:
  PYTHONHOME = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  PYTHONPATH = '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/itk:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/python:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages'
  program name = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  sys.base_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.base_exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.platlibdir = 'lib'
  sys.executable = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  sys.prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.path = [
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/itk',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/python',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python310.zip',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python3.10',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

The outside environment setup a pretty intrusive set of environment variables, how to protect against it?

[Moelf]

current CondaPkg.toml is

  [deps]
  python = ">=3.10,<4"
  uproot = ""
  hist = ""

[Moelf]

ERROR: LoadError: InitError: failed process: Process(setenv(`/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python 
/home/jiling/.julia/packages/PythonCall/eU0yr/src/cpython/find_libpython.py --list-all`,["OpenLoopsPath=/cvmfs/sft.cern.ch
/lcg/releases/MCGenerators/openloops/2.1.2-8fff2/x86_64-centos7-gcc11-opt", "PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views
/dev4/Wed/x86_64-centos7-gcc11-opt/scripts:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/bin:
/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/bin:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-
centos7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/opt/dell/srvadmin/bin:/home/jiling/.fzf/bin:
/home/jiling/.local/bin:/home/jiling/bin", "EDITOR=vim", "LHAPDF_DATA_PATH=/cvmfs/sft.cern.ch/lcg/external/lhapdfsets/current
/:/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/lhapdf/6.5.3-d2254/x86_64-centos7-gcc11-opt/share/LHAPDF:", 
"CXX=/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/bin/g++", "CC=/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-
ad0f5/x86_64-centos7/bin/gcc", "LD_LIBRARY_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/thepeg/2.2.3-9e086
/x86_64-centos7-gcc11-opt/lib/ThePEG:/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/herwig++/7.2.3-54ceb/x86_64-
centos7-gcc11-opt/lib/Herwig:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-
packages/jaxlib/mlir/_mlir_libs:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-
packages/torch/lib:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/onnxruntime/capi/:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/tensorflow:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/tensorflow
/contrib/tensor_forest:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/tensorflow/python/framework:/cvmfs/sft.cern.ch/lcg/releases/java/8u312-80070/x86_64-centos7-gcc11-opt/jre/lib/amd64:/cvmfs
/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib64:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed
/x86_64-centos7-gcc11-opt/lib:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/lib:/cvmfs/sft.cern.ch/lcg/releases
/gcc/11.3.0-ad0f5/x86_64-centos7/lib64:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/lib:/cvmfs/sft-
nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-opt/lib64/R/library/readr/rcon", "GOBIN=/cvmfs/sft-
nightlies.cern.ch/lcg/latest/gophernotes/7c3ca4e-d1acd/x86_64-centos7-gcc11-opt/bin", "DD4hepINSTALL=/cvmfs/sft-
nightlies.cern.ch/lcg/nightlies/dev4/Wed/DD4hep/01.23/x86_64-centos7-gcc11-opt", "MANPATH=/cvmfs/sft-nightlies.cern.ch
/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/man:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-
opt/share/man:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/share/man:/cvmfs/sft.cern.ch/lcg/releases/binutils
/2.37-355ed/x86_64-centos7/share/man::/opt/puppetlabs/puppet/share/man"  …  "GVBINDIR=/cvmfs/sft-nightlies.cern.ch
/lcg/latest/graphviz/2.40.1-0ed86/x86_64-centos7-gcc11-opt/lib/graphviz", "JULIA_LOAD_PATH=/home/jiling/.julia
/dev/WVZAnalysis/Project.toml:/home/jiling/.julia/environments/v1.7/Project.toml:/cvmfs/sft.cern.ch/lcg/releases/julia/1.7.3-
b9498/x86_64-centos7-gcc11-opt/share/julia/stdlib/v1.7:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-
opt/share/julia/environments/v1.7/Project.toml", "PKG_CONFIG_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-
centos7-gcc11-opt/lib64/pkgconfig:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/pkgconfig", 
"ROOT_INCLUDE_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/include/Geant4:/cvmfs
/sft.cern.ch/lcg/releases/jsonmcpp/3.10.5-f26c3/x86_64-centos7-gcc11-opt/include:/cvmfs/sft-nightlies.cern.ch/lcg/views
/dev4/Wed/x86_64-centos7-gcc11-opt/src/cpp:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/include:
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/include/python3.9:/cvmfs/sft-nightlies.cern.ch
/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-opt/lib64/R/include:/cvmfs/sft-nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-
centos7-gcc11-opt/lib64/R/library/RInside/include:/cvmfs/sft-nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-
opt/lib64/R/library/Rcpp/include", "DATA=/data/jiling", "G4ABLADATA=/cvmfs/geant4.cern.ch/share/data/G4ABLA3.1", 
"G4INSTALL=/cvmfs/sft-nightlies.cern.ch/lcg/latest/Geant4/11.0.3-78116/x86_64-centos7-gcc11-opt", 
"OPENBLAS_MAIN_FREE=1", "OCTAVE_HOME=/cvmfs/sft-nightlies.cern.ch/lcg/latest/octave/6.2.0-055df/x86_64-centos7-
gcc11-opt", "GOROOT=/cvmfs/sft.cern.ch/lcg/releases/go/1.12.5-ff8e6/x86_64-centos7-gcc11-opt"]), ProcessExited(1)) [1]

[Moelf]

this works

$ env PYTHONHOME= PYTHONPATH= julia --project=.

[cjdoris]

Ah ok thanks, I need to make CondaPkg clear out any Python env vars.

[Moelf]

can that be done? if so yeah that should work; I was wondering if it's a CondaPkg bug

[cjdoris]

Yep, you can do delete!(ENV, "PYTHONHOME") for example (it only affects the running process). CondaPkg already modifies other env vars.

[Moelf]

that doesn't work, the sub-processes spawned later gets envs from the original Julia process

[cjdoris]

It does, if you do delete!(ENV, "PYTHONHOME") from Julia then start Python from within Julia, it won't see PYTHONHOME.

[Moelf]

sorry, I didn't mention that when I say it doesn't work, I already tried.

[Moelf]

ENV["PYTHONHOME"] = ""
ENV["PYTHONPATH"] = ""
using PythonCall, Serialization

function make_TH1D(h)
    np = pyimport("numpy")
...
end

this fails in the middle of the CondaPkg trying to install the environment and I still see those environment variable

[Moelf]

bump

[Moelf]

I also don't quite understand why this wouldn't work

module A

if contains(ENV["PYTHONHOME"], "cern.ch/lcg/release")
    @info "Using LCG Python, if it doesn't work try delete `.CondaPkg/` folder"
     # use LCG release python
     ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
end
using PythonCall
end

I tested this in a standalone PythonCall.jl it works, but not when used in another package?

[cjdoris]

I can reproduce your issue, and have pushed a fix to main. Please try it out with pkg> add PythonCall#main.

[Moelf]

it still doesn't work, if I have a package that looks like this:

module WVZPythonExt
    function __init__()
    ▏   if contains(ENV["PYTHONHOME"], "cern.ch/lcg/release")
    ▏   @info "Using LCG Python, if it doesn't work try delete `.CondaPkg/` folder"
    ▏   ▏   ▏   # use LCG release python
    ▏   ▏   ▏   ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
    ▏   end
    end

    using PythonCall, FHist, Serialization

    export serial_to_root

    function make_TH1D(h)
    ▏   np = pyimport("numpy")
    ▏   pyhist = pyimport("hist")

when I using this pkg, PythonCall would just try to download conda and does its own thing

[cjdoris]

The __init__() function is not called until after the module is loaded (and PythonCall is already initialised) which is too late. You need to set the env var before loading the package.