Unbelieveble

[GitOguz]

Never in my whole life have I had to install or try to get such a difficult application combination to work. 100,000 different packages and libraries, with prehistoric versions that are not even up to date. It's 2025, and NONMEM still uses MPICH2 1.x.x, which is ancient. Meanwhile, version 4 is available. PsN is left in the dark.

I invest in expensive hardware to run NONMEM jobs with multiple nodes. Each server is dual-socket. To use all its capacity, you need MPI. The complexity of PsN and how it can run multi-node, multi-socket MPI jobs is beyond me. Everything is installed, and MPI tests without NONMEM work fine. But with NONMEM, it’s a complete error cascade. I have no clue where to even begin.

For example, if you have 2 nodes, each with 2 CPUs (2 x 64 = 128 threads per node), giving a total of 256 threads across 2 nodes, how do you even run a parallel job using PsN with a 2-node cluster?

[rikardn]

I feel your frustation 😉. PsN can either run parallel NONMEM using MPI or parallelize across multiple NOMEM jobs (to for example run each NONMEM job of a bootstrap in parallel). For the first option you'd use MPI to connect the nodes and in the second you would need a job scheduler like for example Slurm.

Assuming you intend to use MPI I would first make sure that NONMEM can run without MPI outside of PsN. Then I would run the MPI test (it seems as you already did that). Then create a parafile for your particular setup. You could start from one of the ones provided by NONMEM. Does it seem like NONMEM can run a model in parallel across nodes with the parafile option?

[GitOguz]

When I use sample parallelization file mpilinux_twocomputer.pnm and adjust it so it suits our setup wont work.

Do you have example pnm file for 2 server setup?

[rikardn]

We have only used Slurm for clusters. It should be possible to setup a cluster with only MPI, by configuring it to know about all machines in a local network. I don't think changes to the pnm-file is needed for this.

[GitOguz]

I have also installed slurm on 2 node setup. But dont I still need to use pnm file to able to use MPI for multi socket multi node calculations?

[rikardn]

Yes, you need it, but I don't think any particular changes are needed depending on the number of computers that are connected. The MPI parallelization is a NONMEM feature so please consult the NONMEM user guides for details. In particular check how nmfe can insert templated options into the pnm-file. The PsN -nodes option is inserted this way into the [nodes]