AMPI + Exodus + HDF5 + NetCDF

[jbakosi]

Hi folks,

I managed to build my app that uses an MPI-only lib (ExodusII from SNL) on top of AMPI instead of OpenMPI and here is what I get when I try to run it:

Reason: TCharm has not been initialized!

[0] Stack Traceback:
  [0:0] inciter 0x110aeed CmiAbortHelper(char const*, char const*, char const*, int, int)
  [0:1] inciter 0x1107cd3 
  [0:2] inciter 0xac6b78 TCharm::get()
  [0:3] inciter 0xac2cba TCHARM_Element
  [0:4] inciter 0xac6c52 TCharmAPIRoutine::~TCharmAPIRoutine()
  [0:5] inciter 0xadef3c MPI_Initialized
  [0:6] inciter 0xc10897 H5_init_library
  [0:7] inciter 0xc789e3 H5Eset_auto2
  [0:8] inciter 0xbf727f 
  [0:9] inciter 0xbf7233 nc4_hdf5_initialize
  [0:10] inciter 0xbb4de9 NC_HDF5_initialize
  [0:11] inciter 0xbb5e96 nc_initialize
  [0:12] inciter 0xb85275 NC_open
  [0:13] inciter 0xb851f6 nc_open
  [0:14] libexodus.so 0x72673fd ex_open_int
  [0:15] libxyst_exodusiimeshio.so 0x6fc3ff8 tk::ExodusIIMeshReader::ExodusIIMeshReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
  [0:16] libxyst_inciter.so 0x6b84019 inciter::Transporter::createPartitioner()
  [0:17] libxyst_inciter.so 0x6b856fa inciter::Transporter::Transporter()
  [0:18] libxyst_inciter.so 0x6b8f5ec inciter::CkIndex_Transporter::_call_Transporter_void(void*, void*)
  [0:19] inciter 0xfb7098 CkDeliverMessageFree
  [0:20] inciter 0xfb7841 
  [0:21] inciter 0xfbc550 
  [0:22] inciter 0xfb935e 
  [0:23] inciter 0xfb8aea _processHandler(void*, CkCoreState*)
  [0:24] inciter 0x10c1657 CmiHandleMessage
  [0:25] inciter 0x10c194e CsdScheduleForever
  [0:26] inciter 0x10c16aa CsdScheduler
  [0:27] inciter 0x110ac7e 
  [0:28] inciter 0x110a5a9 ConverseInit
  [0:29] inciter 0x10a809f charm_main
  [0:30] inciter 0xed8362 main
  [0:31] libc.so.6 0x766ad0a __libc_start_main
  [0:32] inciter 0x9c3e5a _start
CHARM++ FATAL ERROR: TCharm has not been initialized!

Caught signal 11 (SIGSEGV)
>>> Exception: Signal caught
>>>
>>> =========== CALL TRACE ===========
>>>
>>> /home/jbakosi/code/xyst/build/clang/Base/libxyst_base.so : tk::signalHandler(int)+0x1ad
>>> /lib/x86_64-linux-gnu/libc.so.6 : ()+0x38d60
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : LrtsAbort(char const*)+0x63
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0x110aef6]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0x1107cd3]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : TCharm::get()+0x28
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : TCHARM_Element()+0x2a
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : TCharmAPIRoutine::~TCharmAPIRoutine()+0x62
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : MPI_Initialized()+0x6c
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : H5_init_library()+0x17
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : H5Eset_auto2()+0x83
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0xbf727f]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : nc4_hdf5_initialize()+0x13
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : NC_HDF5_initialize()+0x29
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : nc_initialize()+0x86
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : NC_open()+0x75
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : nc_open()+0x46
>>> /home/jbakosi/code/xyst/build/clang/exodus/libexodus.so : ex_open_int()+0x1cd
>>> /home/jbakosi/code/xyst/build/clang/IO/libxyst_exodusiimeshio.so : tk::ExodusIIMeshReader::ExodusIIMeshReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x238
>>> /home/jbakosi/code/xyst/build/clang/Inciter/libxyst_inciter.so : inciter::Transporter::createPartitioner()+0xa49
>>> /home/jbakosi/code/xyst/build/clang/Inciter/libxyst_inciter.so : inciter::Transporter::Transporter()+0x9fa
>>> /home/jbakosi/code/xyst/build/clang/Inciter/libxyst_inciter.so : inciter::CkIndex_Transporter::_call_Transporter_void(void*, void*)+0x3c
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : CkDeliverMessageFree()+0x68
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0xfb7841]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0xfbc550]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0xfb935e]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : _processHandler(void*, CkCoreState*)+0x27a
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : CmiHandleMessage()+0x97
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : CsdScheduleForever()+0xce
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : CsdScheduler()+0x1a
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter() [0x110ac7e]
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : ConverseInit()+0x709
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : charm_main()+0x3f
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : main()+0x22
>>> /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xea
>>> /home/jbakosi/code/xyst/build/clang/Main/inciter : _start()+0x2a
>>>
>>> ======= END OF CALL TRACE ========

What does "TCharm has not been initialized!" mean?

Thanks,
Jozsef

[jbakosi]

Forgot to add info on my charm build:

==============================
Charm++ 7.0.0 configuration: 
  OS version:      Debian GNU/Linux 11 (bullseye) (Linux 5.10.0-23-amd64, x86_64)
  Charm++ version: 9640535* (dev)
  Hostname:        sprout
  Machine layer:   netlrts-linux-x86_64
  Build target:    AMPI
  Build type:      Debug
  C compiler:      /usr/lib/ccache/clang [Clang 11.0.1]
  C++ compiler:    /usr/lib/ccache/clang++ [Clang 11.0.1]
  F90 compiler:    /usr/bin/f95 [GNU 10.2.1]
  CMake:           /usr/bin/cmake [3.18.4]
  Charmc flags:    -g
  Enabled options: TRACING ERROR_CHECKING ZLIB 
==============================

[stwhite91]

With AMPI you need to privatize the global variables of all MPI code/libraries to the virtual ranks for correctness when running with multiple ranks per process. HDF5 contains hundreds of mutable global/static variables. In the past we privatized HDF5 using TLSglobals. This requires you to build our modified version of HDF5 using -tlsglobals. You could also try using pieglobals on an unmodified version of HDF5, but that hasn't been tested and I'm not sure how pieglobals would interact with AMPI + Charm++ interoperation. In general, AMPI + Charm++ interoperability is experimental and not well tested at this point.

[jbakosi]

Thanks, Sam. That is useful to know. I did find and have used in the past the modified hdf5 you mention. While looking into this in more detail, however, I realized that my app does not even need hdf5 that importantly, as for the functionality we use, we can get away with relying on the serial version of netcdf within the exodus library to read/write unstructured meshes and associated data in parallel. So this problem goes away.