Parallel efficiency issue

[Jiewang11]

Details

在LSF超算系统上，对135原子体系进行结构优化计算
下面的附件处无法使用，暂时无法上传具体文件
调用了24核。48核，72核分别进行计算，但是计算速度都约是20秒/电子步
尝试使用kapr参数，但是基本没有多少提速效果。
难道对于这样的体系，使用24核计算就已经达到了计算的上限速度了吗？

部分计算参数如下

System variables

calculation relax
symmetry 0
kspacing 0.14 # unit in 1/bohr
precision double # or single
kpar 2
vdw_method d3_0
dft_functional PBE
basis_type lcao

超算提交脚本如下
#!/bin/bash
#BSUB -J abacus_test2
#BSUB -q mpi
#BSUB -n 24
#BSUB -o job.out
#BSUB -e job.err

加载环境

#source /share/home/xswu/wangjie/abacus/abacus_load.sh
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

并行运行

mpirun abacus > log

这里mpirun abacus > log不能使用mpirun -np 24 abacus > log,否则就会报错
prterun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 24; it may have occurred for other processes as well. NOTE: A common cause for this error is misspelling a prterun command line parameter option (remember that prterun interprets the first unrecognized command line token as the executable). Node: c60n11 Executable: /share/home/xswu/wangjie/abacus/abacus-develop-LTS/bin/abacus

Task list for Issue attackers (only for developers)

• Reproduce the performance issue on a similar system or environment.
• Identify the specific section of the code causing the performance issue.
• Investigate the issue and determine the root cause.
• Research best practices and potential solutions for the identified performance issue.
• Implement the chosen solution to address the performance issue.
• Test the implemented solution to ensure it improves performance without introducing new issues.
• Optimize the solution if necessary, considering trade-offs between performance and other factors (e.g., code complexity, readability, maintainability).
• Review and incorporate any relevant feedback from users or developers.
• Merge the improved solution into the main codebase and notify the issue reporter.

[QuantumMisaka]

@Jiewang11 Thanks for you issue! could you please upload your STRU for test? The problem you reported is seems to related to your HPC environments and ABACUS installation method, you can tell us more information so that we can have better understanding. Thanks !
Also it is welcomed to join ABACUS QQ Group for instant communication with ABACUS running problem: 759914681

[Jiewang11]

我大概弄清楚问题所在，编译的时候，除了gcc是超算加载的，其余依赖全是下载的。编译成之后，跨节点运行没有成功调用其他节点的openmpi。只能以24核（一个几点）运行。
后来加载系统的其他几个组件，包括openmpi，重新进行了编译。结果是可以成功运行了，并且使用多节点。但是节点之间的通信时一个问题。ucx需要1.11版本以上，但是超算只有1.7.调用不了ucx。只能退而求其次使用openib,48核比24核差不多快1倍。但是72核比48核就快了一点。