Investigate packaging genie_python [timebox: 2.5 days]

[zsoltkebel]

This Ticket should instead be a discussion based on points raised in comments

I suggested investigating whether we could turn genie_python helper functions into a pip installable package from vcs for better maintenance and clarity.

I suggest revisiting this ticket in planning and deciding priority etc.

Work done

On branch ticket8381-package in the genie_python repository.

• ticket branch created and requirements.txt file trimmed to the package that are used by genie_python code. We had a lot of seemingly unused dependencies (pygame?), some of which were severely outdated (e.g. https://pypi.org/project/ipython_genutils/)
  • pygame might be used for collision avoidance on some instruments. not in genie python itself
• Note that some environment variables have to be set for the CaChannel package to install correctly through pip.

Going further

• Dependencies seemingly unused here might be necessary on instruments, somehow separate this. Idially we don't want genie_python to list any dependency that it doesn't use directly.
• Do we need caRepeater.exe in our Python installation directory? One of the install steps is moving this into there. We already have one at EPICS dir. do we need both? Could not find usage of it within genie_python code.
• Perhaps separate the python install from genie_python helper functions (package)

discussed ~1:21 20/60/24/

planning 00:00 2024/10/31

[FreddieAkeroyd]

Should we consider splitting some parts of genie python that are server specific and/or shared into separate packages e.g. i think we have things like CaChannel wrappers that may be common to both client and server, but scripting stuff is not needed by inst_servers, which could run in their own virtual environment separate from genie python

[FreddieAkeroyd]

regarding caRepeater, it is launched by the EPICS ca.dll which needs to find it in your PATH. As the epics site packages is not in the PATH, i think it got copied to next to python.exe so it would be found. However we now build cachannel with our own epics that i modified to add the directory where ca.dll is loaded from to the internal search path, so it would probably work without it next to python.exe now

[Tom-Willemsen]

This likely needs splitting into a series of tickets - I think this is too big to just do as one ticket.

Some likely subtasks which I think we need/want to do as part of this (in no particular order) - each of which could be a sensible ticket in itself:

• Ensure genie_python can be made public (may need some git history manipulation)
  • See discussion in teams "technical" channel.
• Create an isiscomputinggroup or similar organisation in pypi
  • Or we could use a private pypi server, but for sharing with other groups I think we should prefer to put everything on pypi directly
• Decide our python version support policy
  • What we currently do: just one version, but that's not great for sharing with other groups and limits the benefit that we get from packaging genie_python
  • "Industry standard" for scientific python is SPEC00 which typically covers 3 recent python versions
  • If we need to support old(er) linux distributions e.g. RHEL, that may mean we need to support quite old python versions
  • Something else.
  • n.b. ask mantid, idaas etc what their python version support policy is for comparison
  • And then actually make genie_python follow it & run at least the unit tests against those versions.
• Work out what to do with our customized builds - may need to build and package these ourselves, or contribute our changes upstream:
  • EPICS libraries - complexity as we use local-ish builds - was mostly done in Use epicscorelibs for python modules #8453
  • smslib - currently has a complex build, but complexity could be removed relatively easily. Ask in technical channel on teams if unsure what this means. Genie_python: Use server to send alerts #5570 .
  • ode - we build our own wheels currently. However it's only actually used by collision avoidance monitor, which in turn isn't being used in practice. Best solution may be to remove ODE as a genie_python dependency, and add it as a dependency of a CAM venv explicitly, installing a wheel from the share.
  • readline - we apply a manual patch currently. Our dependency is quite outdated. We may need to search for a compatible, supported alternative. Might no longer be required at all. e.g. https://github.com/pyreadline3/pyreadline3 may help.
  • Ensure these are all installable as "standard" python modules, not using manual robocopies / env variables / relying on compilers or EPICS being present etc
• Ensure our builds & dependencies are .bat-script (and .sh) free, using standard cross-platform python packaging mechanisms not manual robocopies and environment variable sets etc.
• Separate library from distribution.
  • The end result of installing the distribution should look identical to currently, but the sources should reside in two separate repositories - one for the library and one for the distribution.
  • The "library" repo should use the src layout as specified here for it's repository structure.
  • The requirements of the library repo should only be what the genie_python library itself uses
  • The "distribution" repo should contain the requirements.txt for requirements not directly needed by the genie_python library (e.g. things like scipy which we provide for users, but do not directly depend on)
  • Give library and distribution different names so that we don't get confused when we talk about "genie_python" whether we're talking about library or distribution.
• Move to PEP621 (pyproject.toml) for genie_python
  • See ibex_bluesky_core for an example
  • Use setuptools_scm to auto-generate version metadata from git tags as per i_b_c
• Set up build to automatically push new releases of genie_python to pypi
  • Follow pattern ibex_bluesky_core uses for this
• Generate tickets to start making installable packages for other modules (e.g. IocTestFramework) would be a good candidate so that we can use those as dependencies of individual support modules
• Thin out dependencies of the distribution, in principle leaving only things that scientists will want to use. Our code should move towards .venvs, into which we can install whatever we need without making it a global dependency of the distribution
• Sort out genie_python's unit tests to consistently not try to do i/o , and especially not i/o on module import
  • This will dramatically speed up unit tests
• In distribution, get rid of contextlib2 and other python-2 only libraries (after checking/removing usage in other repos)
• Make genie_python's docs better, they exist but are extremely minimal.
• Move distribution creation towards a better mechanism. Environment management mechanism uktena#2