included ROMIO Intel GPU (ZE) doesn't build

[jeffhammond]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

fdf50d0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Building from source

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

$ git submodule status
 10fe4735ee374f5807c2160e61274c4aa53491ae ../3rd-party/openpmix (v1.1.3-3847-g10fe4735)
 d8bd12b3ffda4af6918d641f024a6b0118789700 ../3rd-party/prrte (psrvr-v2.0.0rc1-4624-gd8bd12b3ff)
 c1cfc910d92af43f8c27807a9a84c9c13f4fbc65 ../config/oac (remotes/origin/HEAD)

Please describe the system on which you are running

• Operating system/version: Linux nuclear 5.19.0-41-generic #42~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 18 17:40:00 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• Computer hardware: 11th Gen Intel(R) Core(TM) i7-1165G7
• Network type: None

---

Details of the problem

../configure --disable-debug-symbols --without-mpi-param-check --with-ucx --prefix=/opt/ompi/gcc-ucx-fast --with-pmix=internal && make -j4

With MPICH, I use --without-ze because of all the problems that Intel GPU support causes but I can't figure out how to cause this in Open-MPI ROMIO.

  CC       src/shm/mpl_shm.lo
  CC       src/shm/mpl_shm_sysv.lo
  CC       src/shm/mpl_shm_mmap.lo
  CC       src/shm/mpl_shm_win.lo
  CC       src/gpu/mpl_gpu_common.lo
  CC       src/gpu/mpl_gpu_ze.lo
  CC       src/gavl/mpl_gavl.lo
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_ipc_handle_create’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:123:11: warning: implicit declaration of function ‘zeDriverGetMemIpcHandle’; did you mean ‘zeMemGetIpcHandle’? [-Wimplicit-function-declaration]
  123 |     ret = zeDriverGetMemIpcHandle(global_ze_driver_handle, ptr, ipc_handle);
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           zeMemGetIpcHandle
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_ipc_handle_map’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:139:9: warning: implicit declaration of function ‘zeDriverOpenMemIpcHandle’; did you mean ‘zeMemOpenIpcHandle’? [-Wimplicit-function-declaration]
  139 |         zeDriverOpenMemIpcHandle(global_ze_driver_handle,
      |         ^~~~~~~~~~~~~~~~~~~~~~~~
      |         zeMemOpenIpcHandle
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:140:69: error: ‘MPL_gpu_ipc_mem_handle_t’ {aka ‘struct _ze_ipc_mem_handle_t’} has no member named ‘global_dev_id’
  140 |                                  global_ze_devices_handle[ipc_handle.global_dev_id],
      |                                                                     ^
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:141:44: error: ‘MPL_gpu_ipc_mem_handle_t’ {aka ‘struct _ze_ipc_mem_handle_t’} has no member named ‘handle’
  141 |                                  ipc_handle.handle, ZE_IPC_MEMORY_FLAG_NONE, ptr);
      |                                            ^
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:141:53: error: ‘ZE_IPC_MEMORY_FLAG_NONE’ undeclared (first use in this function); did you mean ‘ZE_IPC_MEMORY_FLAG_FORCE_UINT32’?
  141 |                                  ipc_handle.handle, ZE_IPC_MEMORY_FLAG_NONE, ptr);
      |                                                     ^~~~~~~~~~~~~~~~~~~~~~~
      |                                                     ZE_IPC_MEMORY_FLAG_FORCE_UINT32
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:141:53: note: each undeclared identifier is reported only once for each function it appears in
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_ipc_handle_unmap’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:156:11: warning: implicit declaration of function ‘zeDriverCloseMemIpcHandle’; did you mean ‘zeMemCloseIpcHandle’? [-Wimplicit-function-declaration]
  156 |     ret = zeDriverCloseMemIpcHandle(global_ze_driver_handle, ptr);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~
      |           zeMemCloseIpcHandle
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_query_pointer_attr’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:171:11: warning: implicit declaration of function ‘zeDriverGetMemAllocProperties’; did you mean ‘zeDriverGetIpcProperties’? [-Wimplicit-function-declaration]
  171 |     ret = zeDriverGetMemAllocProperties(global_ze_driver_handle, ptr, &ptr_attr, &device);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |           zeDriverGetIpcProperties
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_malloc’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:202:25: error: ‘ZE_DEVICE_MEM_ALLOC_FLAG_DEFAULT’ undeclared (first use in this function); did you mean ‘ZE_DEVICE_MEM_ALLOC_FLAG_FORCE_UINT32’?
  202 |     device_desc.flags = ZE_DEVICE_MEM_ALLOC_FLAG_DEFAULT;
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                         ZE_DEVICE_MEM_ALLOC_FLAG_FORCE_UINT32
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:204:16: error: ‘ze_device_mem_alloc_desc_t’ {aka ‘struct _ze_device_mem_alloc_desc_t’} has no member named ‘version’
  204 |     device_desc.version = ZE_DEVICE_MEM_ALLOC_DESC_VERSION_CURRENT;
      |                ^
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:204:27: error: ‘ZE_DEVICE_MEM_ALLOC_DESC_VERSION_CURRENT’ undeclared (first use in this function); did you mean ‘ZE_DEVICE_LUID_EXT_VERSION_CURRENT’?
  204 |     device_desc.version = ZE_DEVICE_MEM_ALLOC_DESC_VERSION_CURRENT;
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                           ZE_DEVICE_LUID_EXT_VERSION_CURRENT
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:208:11: warning: implicit declaration of function ‘zeDriverAllocDeviceMem’; did you mean ‘zeMemAllocDevice’? [-Wimplicit-function-declaration]
  208 |     ret = zeDriverAllocDeviceMem(global_ze_driver_handle, &device_desc,
      |           ^~~~~~~~~~~~~~~~~~~~~~
      |           zeMemAllocDevice
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_malloc_host’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:223:23: error: ‘ZE_HOST_MEM_ALLOC_FLAG_DEFAULT’ undeclared (first use in this function); did you mean ‘ZE_HOST_MEM_ALLOC_FLAG_FORCE_UINT32’?
  223 |     host_desc.flags = ZE_HOST_MEM_ALLOC_FLAG_DEFAULT;
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                       ZE_HOST_MEM_ALLOC_FLAG_FORCE_UINT32
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:224:14: error: ‘ze_host_mem_alloc_desc_t’ {aka ‘struct _ze_host_mem_alloc_desc_t’} has no member named ‘version’
  224 |     host_desc.version = ZE_HOST_MEM_ALLOC_DESC_VERSION_CURRENT;
      |              ^
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:224:25: error: ‘ZE_HOST_MEM_ALLOC_DESC_VERSION_CURRENT’ undeclared (first use in this function)
  224 |     host_desc.version = ZE_HOST_MEM_ALLOC_DESC_VERSION_CURRENT;
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:229:11: warning: implicit declaration of function ‘zeDriverAllocHostMem’ [-Wimplicit-function-declaration]
  229 |     ret = zeDriverAllocHostMem(global_ze_driver_handle, &host_desc, size, mem_alignment, ptr);
      |           ^~~~~~~~~~~~~~~~~~~~
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_free’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:240:11: warning: implicit declaration of function ‘zeDriverFreeMem’ [-Wimplicit-function-declaration]
  240 |     ret = zeDriverFreeMem(global_ze_driver_handle, ptr);
      |           ^~~~~~~~~~~~~~~
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c: In function ‘MPL_gpu_get_dev_handle’:
../../../../3rd-party/romio341/mpl/src/gpu/mpl_gpu_ze.c:280:19: error: ‘device_handles’ undeclared (first use in this function); did you mean ‘dev_handle’?
  280 |     *dev_handle = device_handles[dev_id];
      |                   ^~~~~~~~~~~~~~
      |                   dev_handle
make[3]: *** [Makefile:1158: src/gpu/mpl_gpu_ze.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/home/jhammond/MPI/ompi/buildd/3rd-party/romio341/mpl'
make[2]: *** [Makefile:4709: all-recursive] Error 1
make[2]: Leaving directory '/home/jhammond/MPI/ompi/buildd/3rd-party/romio341'
make[1]: *** [Makefile:1403: all-recursive] Error 1
make[1]: Leaving directory '/home/jhammond/MPI/ompi/buildd/3rd-party'
make: *** [Makefile:1530: all-recursive] Error 1

[ggouaillardet]

you should be able to build Open MPI without ROMIO with configure --disable-io-romio
and Open MPI without ze with configure --with-io-romio-flags=--without-ze

(I wrote should because this is not something I have tested recently)

[jeffhammond]

Yes, I did the former because I don't use MPI IO. I pondered the latter but didn't try it.

[intel-hm]

tried today to compile 5.0.6 on SLES 15.5 with this configure line:
./configure --prefix=/lfs/lfs19/mhebenst/openmpi/5.0.6 --with-libfabric=/lfs/lfs19/mhebenst/openmpi/5.0.6 --with-ucx=/lfs/lfs19/mhebenst/openmpi/5.0.6 --disable-cuda --with-io-romio-flags=--without-ze

and got the same error. disabling romio completely worked

[intel-hm]

I used this patch to remove ze - seems to work, OSU test runs gave expected performance:

mhebenst@ecompile13:workloads>cat ../../openmpi/mpl_remove_zesupport.patch
--- openmpi-5.0.6/3rd-party/romio341/mpl/Makefile.in    2024-11-15 07:20:30.000000000 -0700
+++ openmpi-5.0.6_v1/3rd-party/romio341/mpl/Makefile.in 2025-01-10 11:42:43.000000000 -0700
@@ -90,8 +90,7 @@
 build_triplet = @build@
 host_triplet = @host@
 @MPL_HAVE_CUDA_TRUE@am__append_1 = src/gpu/mpl_gpu_cuda.c
-@MPL_HAVE_CUDA_FALSE@@MPL_HAVE_ZE_TRUE@am__append_2 = src/gpu/mpl_gpu_ze.c
-@MPL_HAVE_CUDA_FALSE@@MPL_HAVE_ZE_FALSE@am__append_3 = src/gpu/mpl_gpu_fallback.c
+@MPL_HAVE_CUDA_FALSE@am__append_3 = src/gpu/mpl_gpu_fallback.c
 TESTS = $(am__EXEEXT_1)
 check_PROGRAMS = $(am__EXEEXT_1)
 @MPL_EMBEDDED_MODE_TRUE@am__append_4 = $(mpl_headers)
@@ -178,9 +177,7 @@
        src/gpu/mpl_gpu_fallback.c src/gavl/mpl_gavl.c
 am__dirstamp = $(am__leading_dot)dirstamp
 @MPL_HAVE_CUDA_TRUE@am__objects_1 = src/gpu/mpl_gpu_cuda.lo
-@MPL_HAVE_CUDA_FALSE@@MPL_HAVE_ZE_TRUE@am__objects_2 =  \
-@MPL_HAVE_CUDA_FALSE@@MPL_HAVE_ZE_TRUE@        src/gpu/mpl_gpu_ze.lo
-@MPL_HAVE_CUDA_FALSE@@MPL_HAVE_ZE_FALSE@am__objects_3 = src/gpu/mpl_gpu_fallback.lo
+@MPL_HAVE_CUDA_FALSE@am__objects_3 = src/gpu/mpl_gpu_fallback.lo
 am_lib@MPLLIBNAME@_la_OBJECTS = src/atomic/mpl_atomic.lo \
        src/bt/mpl_bt.lo src/dbg/mpl_dbg.lo src/env/mpl_env.lo \
        src/mem/mpl_trmem.lo src/msg/mpl_msg.lo src/sock/mpl_sock.lo \