NumPy 2

[DanielYang59]

I noticed there is a branch to drop numpy<2 pin, any progress on this?

• pynxtools-apm: https://github.com/FAIRmat-NFDI/pynxtools-apm/blob/main/pyproject.toml#L26
• pynxtools-em https://github.com/FAIRmat-NFDI/pynxtools-em/blob/main/pyproject.toml#L30
• pynxtools-xps https://github.com/FAIRmat-NFDI/pynxtools-xps/blob/main/pyproject.toml#L27

[lukaspie]

Hey @DanielYang59, we didn't make much progress on this because NOMAD always had numpy<2 and we mostly try to keep the same dependencies as NOMAD. Is this a planned change in NOMAD as well? For most of pynxtools and its plugin, it should be fairly simple to unpin.

[DanielYang59]

Yes we're trying to remove np<2 and test plugins and found quite a few pynxtools related packages still have this constraint. cc @blueraft

[blueraft]

Looks like in pynxtools the upper bound can be safely removed, even if nomad-lab still enforces numpy<2.

I just looked at the pynxtools-em numpy-v2 branch but seeems like some unrelated things are also happening there.

[lukaspie]

Removal of the upper bound is unfortunately not completely trivial because of https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/hdfdict.py#L10. This is a functionality we copied from another repo because there wasn't a good versioning in place in the original code. Replacing numpy.string_ with numpy.bytes_ as suggested by the ruff extension doesn't help (I guess that is expected) because there are further problems downstreams. Will investigate..

[blueraft]

Does something like this work? (I haven't tested it)

from importlib.metadata import version

np_v2 = int(version('numpy').split('.')[0]) < 2 
if np_v2:
    from numpy import bytes_ as NUMPY_TYPE
else:
    from numpy import string_ as NUMPY_TYPE

[lukaspie]

We can probably go with master...support-npv2. I'll test it tomorrow a bit more, we'll also have to unpin in all the pynxtools-* packages as @DanielYang59 suggested. Note that the initial branch (npyv2) will not be changed as this is currently used by a collaborator and expects a special set of NeXus definitions that is different to the main branch here.

[mkuehbach]

For pynxtools-em and -apm going with "numpy" without upper bound constraint is desirable and possible. Having this implemented for all pynxtools plugins though was not planned to become included already in the 1.3.17 or was it ?

[lukaspie]

No, this is not part of 1.3.17, so not immediately needed. We can aim to finish this after we have new releases of pynxtools that are compatible with v1. 3.17.

[DanielYang59]

Does something like this work? (I haven't tested it)

from importlib.metadata import version

np_v2 = int(version('numpy').split('.')[0]) < 2
if np_v2:
from numpy import bytes_ as NUMPY_TYPE
else:
from numpy import string_ as NUMPY_TYPE

Must be a typo here:

- np_v2 = int(version('numpy').split('.')[0]) < 2 
+ np_v2 = int(version('numpy').split('.')[0]) >= 2 

Numpy has another suggestion for version-dependent code:

if np.lib.NumpyVersion(np.__version__) >= '2.0.0':
    # np2
else:
    # np1

[lukaspie]

I have two draft PRs now unpinning numpy:

• unpin numpy to enable working with numpy version 2 as well #689 -> we don't even need this if-clause for the different versions if we just don't use numpy.string_ anymore
• unpin numpy pynxtools-xps#142 -> this one fails linting because it depends on the last pynxtools release that still has numpy pint

Note that we will probably first have new releases of pynxtools and pynxtools-apm/em this week (to have working versions for the NOMAD RC) using the pinned versions before we'll go and update numpy everywhere.