Rankfile and MPMD Issue

[JaredCrean2]

Background information

I am trying to launch an MPMD job using a rankfile that uses relative indexing (+n0 for the first hostname, +n1 for the second etc.), on a system that uses the SLURM job scheduler

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

OpenMPI v4.1.4.
sbatch --version report 23.02.7

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Not certain (the system administrators installed it), but I looks like it was compiled from source.

Please describe the system on which you are running

• Operating system/version: RHEL8
• Computer hardware: Each node has 2 x Intel Broadwell E5-2695 v4 (18 cores), 128 GB RAM
• Network type: Intel OmniPath

---

Details of the problem

When I try to launch an MPMD program using mpiexec -rf rankfile.txt to specify the layout of the ranks, I get an error if:

• the rankfile uses relative indexing for the hostnames (+n0, +n1 etc.)
  • Replacing the relative indexing with actual hostnames makes the problem go away
• The second application is on a different host that then first
• There are enough ranks
  • The attached example fills up 2 nodes, the first node with 36 ranks the first application, the second node with 36 ranks of the second application. Putting 1 rank of application 1 on the first node and 1 rank of application 2 on the second node does not produce an error

The error I get is:

--------------------------------------------------------------------------
Rankfile claimed host +n1 by index that is bigger than number of allocated hosts.
--------------------------------------------------------------------------
[ec534:05640] [[7352,0],0] ORTE_ERROR_LOG: Bad parameter in file rmaps_rank_file.c at line 271
[ec534:05640] [[7352,0],0] ORTE_ERROR_LOG: Bad parameter in file base/rmaps_base_map_job.c at line 402

I did allocate 2 nodes, so +n1 should have been a valid hostname.

A minimal reproducer is attached. To run it, do sbatch ./batch_reduced.sh.

As noted in batch_reduced.sh, running 72 ranks of a single application works fine, so this is somehow related to running 2 applications with 36 ranks each.
mpirun_error.tar.gz

[benmenadue]

We just ran into this issue, and have traced it to the mapper generating a new node list for each subsequent app:

ompi/orte/mca/rmaps/rank_file/rmaps_rank_file.c

Lines 184 to 192 in 7256c81

	/* for each app_context, we have to get the list of nodes that it can
	* use since that can now be modified with a hostfile and/or -host
	* option
	*/
	if(ORTE_SUCCESS != (rc = orte_rmaps_base_get_target_nodes(&node_list, &num_slots, app,
	map->mapping, initial_map, false))) {
	ORTE_ERROR_LOG(rc);
	goto error;
	}

Each subsequently generated node list excludes any nodes that then have no slots left:

ompi/orte/mca/rmaps/base/rmaps_base_support_fns.c

Lines 460 to 469 in 7256c81

	/** check to see if this node is fully used - remove if so */
	if (0 != node->slots_max && node->slots_inuse > node->slots_max) {
	OPAL_OUTPUT_VERBOSE((5, orte_rmaps_base_framework.framework_output,
	"%s Removing node %s: max %d inuse %d",
	ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
	node->name, node->slots_max, node->slots_inuse));
	opal_list_remove_item(allocated_nodes, &node->super);
	OBJ_RELEASE(node); /* "un-retain" it */
	continue;
	}

and as a result the relative index from the rankfile is then misinterpreted (it's relative to the remaining nodes rather than the original list).

You get that error once you've mapped more than half your nodes (since any node reference after that is guaranteed to be beyond the end of the list), but even before that any ranks for subsequent apps will be misplaced if you've already completely filled any node.

It's still possible to use the relative node index approach, you just need to take this into account (i.e. +n2 becomes +n1 if node 1 has been filled by an earlier app). But I'm pretty sure this is not the intended behaviour. Not sure how to fix it though.

[rhc54]

Try openpmix/prrte#2052 and see if it helps.