Error: unknown option "-np"

[amaslenn]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

OMPI v3.1.0a1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

It was built using from source using following cmd:

./autogen.sh
./configure --prefix=/path/ompi-v3.1.x
    --with-knem=<path-to-knem>
    --with-mxm=<path-to-mxm>
    --with-hcoll=<path-to-hcoll>
    --with-ucx=<path-to-ucx>
    --with-platform=contrib/platform/mellanox/optimized
    --with-slurm
    --with-pmi

Please describe the system on which you are running

• Operating system/version: RedHat 7.2
• Computer hardware: ConnectX-4 on x86 arch
• Network type: IB

---

Details of the problem

I run tests with MTT and sometimes see a very strange error:
$ /path/ompi-v3.1.x/bin/mpirun -np 3 --debug-daemons --display-map -mca btl_openib_warn_default_gid_prefix 0 --bind-to core --tag-output --timestamp-output -mca pml ucx -x UCX_NET_DEVICES=mlx5_4:1 -mca btl_openib_if_include mlx5_4:1 -mca coll '^hcoll' -x UCX_IB_GID_INDEX=0 -mca osc ucx -x UCX_TLS=rc,sm --map-by node test-app

/path/ompi-v3.1.x/bin/mpirun: Error: unknown option "-np"

It is not reproducible manually, it happens with different test apps and options.

Maybe someone has a clue why it happens.

[amaslenn]

+ @miked-mellanox @alinask

[rhc54]

Check your LD_LIBRARY_PATH and see if the mca_schizo_ompi.la library is present and is correctly being loaded.

[amaslenn]

@rhc54 thanks! Will check it.

[ezze]

@amaslenn, have you solved the issue?

[amaslenn]

No, not yet.

[hppritcha]

@amaslenn has someone from Mellanox helped you with this?

[amaslenn]

@hppritcha yes. I myself work for Mellanox :)

See the issue less often for OMPI@master.

[artpol84]

I have this issue with OMPI master.
@amaslenn have you figured out the way to solve this?

[artpol84]

In my case, it was a conflict of OMPI installation from modules and my own.

[amaslenn]

@artpol84 no, I didn't solve it yet. Happens rarely and not reproducible manually (I didn't succeed with it).
Do you have a manual reproduction?

[jsquyres]

FWIW, the only times I have seen this happen are:

1. You are accidentally not running the mpirun that you think you are. I.e., you're running an mpirun that does not accept -np. The message you sent could be an Open MPI error message, though, and I think Open MPI has always accepted -np, so if it's an Open MPI mpirun, this doesn't shed much light on what could be happening.
2. You accidentally use a - that is not the normal - (e.g., a non-ASCII long-dash or something).
3. There's an accidental non-shell invocation that includes spaces in tokens. Doesn't look like this is the case here (because we would have seen a different error message), but I thought I'd list this for completeness.

[artpol84]

@amaslenn, as I reported in my case the issue was in the conflict of OMPI installations.

I was building my OMPI from sources while having a module with hpcx loaded. This caused schizo components to be linked with libopen-rte.so from modules.
@jsquyres @rhc54 is this an expected behavior?

schizo framework adjusts to the environment. In particular, if I understand correctly it has (directly or indirectly) a list of parameters that are supported by mpirun.
Because of the linking conflict schizo component is unable to load thus only a few parameters are available and "-np" is not.
Actually, if you will take your original cmdline:

/path/ompi-v3.1.x/bin/mpirun -np 3 --debug-daemons --display-map -mca btl_openib_warn_default_gid_prefix 0 --bind-to core --tag-output --timestamp-output -mca pml ucx -x UCX_NET_DEVICES=mlx5_4:1 -mca btl_openib_if_include mlx5_4:1 -mca coll '^hcoll' -x UCX_IB_GID_INDEX=0 -mca osc ucx -x UCX_TLS=rc,sm --map-by node test-app

and switch -np and --debug-daemons (or any other option) you will get mpirun complaining about the very first one it sees. This is because non of them are known as the result of schizo component failure.

The problem is that mpirun is failing silently with schizo and it is very confusing.
We can have a call your evening if you want more details.

[jsquyres]

I was building my OMPI from sources while having a module with hpcx loaded. This caused schizo components to be linked with libopen-rte.so from modules.

I'm not sure what this means -- what are HPCX modules? What is the libopen-rte.so "from modules"?