2025.08.000: ERROR failed to initialize sharp on Gadi

[minghangli-uni]

When running with 2025.08.000, there are repeated error messages in the logs.

• access-om3.out

   1 [gadi-cpu-spr-0022:1851:2041260 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
   2 [gadi-cpu-spr-0002:0:2262497 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
   3 [gadi-cpu-spr-0002:1:2262496 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
   4 [gadi-cpu-spr-0002:2:2262498 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
   5 [gadi-cpu-spr-0002:3:2262499 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
   6 [gadi-cpu-spr-0002:4:2262500 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
...
1976 [gadi-cpu-spr-0017:1337:2596762 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp

1976 is the cpu core count.

• access-om3.err

Failed to open log file (stdout)Failed to open log file (stdout)Failed to open log file (stdout)Failed to open log file (stdout)Failed to open     log file (stdout)Failed to open log file (stdout)Failed to open log file (stdout)Failed to open log file (stdout)Failed to open log file (stdou    t)Failed to open log file (stdout)Failed to open log file ...

The simulation continues and does not crash but with the logs flooded with the above error messages. Wonder if @anton-seaice @dougiesquire you have seen this before releasing it?

[dougiesquire]

I haven't tried to run it (I can now). It's possibly just a Gadi lapse - have you tried rerunning? Is there any more info in the logs about where the issue is coming from?

[minghangli-uni]

have you tried rerunning?

Yes - I submitted two separate jobs and both produced the same error messages.

Is there any more info in the logs about where the issue is coming from?

Unfortunately no. After 1976 [gadi-cpu-spr-0017:1337:2596762 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp, the logs continue with the usual fms messages

1976 [gadi-cpu-spr-0017:1337:2596762 - context.c:307][2025-09-01 16:00:53] ERROR failed to initialize sharp
1977 NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
1978 NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to  5000000.
1979 NOTE from PE     0: field_manager_init:: You are using the legacy version of the field_table

[minghangli-uni]

The above error messages do not appear with 2025.05.001

[anton-seaice]

The run i did with prerelease (access-om3/pr138-15) didn't have the messages.

I think in the past these have been a bit transient. Once it was an issue with the openmpi build which the help desk fixed. If the message doesn't go away then we should email the help desk or @benmenadue

[minghangli-uni]

thanks @anton-seaice. The error messages match exactly with those in the link. I’m just not sure why they’re appearing in this build, since we’re using the same openmpi version (4.1.7). But yes, this is probably a question for @benmenadue

[benmenadue]

I'm not sure why you would be getting that error message, the SHARP-based components are explicitly disabled on Gadi as it's not available there. I just included them in the builds for potential future use (i.e. if we do enable that feature in the network in the future then we can just not disable those components and it will work).

As long as you're using our builds and modules, you shouldn't see those messages unless you're overriding our settings. The issue referenced by @anton-seaice was because I had forgotten to add the setting to disable that component to the module - but that's long since fixed. If you're still seeing that message, can you send through details (e.g. location of job script etc) to help and we can take a look.

[anton-seaice]

the SHARP-based components are explicitly disabled on Gadi as it's not available there

Does this mean it's in the build but disables in the module ?

[minghangli-uni]

The run is located at /g/data/tm70/ml0072/COMMON/git_repos/access-experiment-generator/2025.08.000/c1_parallel_restart_checks/perturb_1/MC_25km_jra_ryf. Job details can be found in the same directory.

The error messages are available at:

• /g/data/tm70/ml0072/COMMON/git_repos/access-experiment-generator/2025.08.000/c1_parallel_restart_checks/perturb_1/MC_25km_jra_ryf/archive/output000/access-om3.out
• /g/data/tm70/ml0072/COMMON/git_repos/access-experiment-generator/2025.08.000/c1_parallel_restart_checks/perturb_1/MC_25km_jra_ryf/archive/output000/access-om3.err

[anton-seaice]

As long as you're using our builds and modules

Oh so this might be the issue ?

$ module use /g/data/vk83/modules/
$ module load access-om3/2025.08.000 
Loading access-om3/2025.08.000
  Loading requirement: openmpi/4.1.7-alvuihq access3-share/2025.08.000-4addtwr access-cice/CICE6.6.1-0-rmyd5d5 access-mom6/2025.07.000-zeetsan access-ww3/2025.08.000-7fuyev2 access3/2025.08.000-ujpt2mh

In the system openmpi/4.1.7 module, sharp is disabled

$ echo $UCC_TLS
^sharp

In openmpi/4.1.7-alvuihq it's not.

Do you have thoughts @aidanheerdegen / @harshula ?

[benmenadue]

@anton-seaice yep, that's likely exactly the problem. You need to use our system modules - they don't just set paths, but they also apply runtime configuration. And that configuration could change in the future e.g. if we are able to enable SHARP support in the network. It's also different depending on where you load the module - e.g. if you're in a job you'll get different settings.

[harshula]

Hi @anton-seaice , Looks like system openmpi should have been used. $PATH contains /apps/openmpi/4.1.7/bin.

$ module use /g/data/vk83/modules/
$ module load access-om3/2025.08.000
Loading access-om3/2025.08.000
  Loading requirement: openmpi/4.1.7-alvuihq access3-share/2025.08.000-4addtwr
    access-cice/CICE6.6.1-0-rmyd5d5 access-mom6/2025.07.000-zeetsan
    access-ww3/2025.08.000-7fuyev2 access3/2025.08.000-ujpt2mh
$ echo $PATH | grep openmpi
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.2.0/access3-2025.08.000-ujpt2mhbhphfgn6qcsyeynoe7lnt2aae/bin:/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.2.0/access-ww3-2025.08.000-7fuyev2pllb5ofejpawwrqofj554rzny/bin:/apps/openmpi/4.1.7/bin:/home/189/hj1810/.local/bin:/home/189/hj1810/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin

[anton-seaice]

Yeah agreed its the same binary - the suggestion is that the environment variables set in in the system openmpi module are different to those in openmpi/4.1.7-alvuihq, which leads to this extraneous logging

[minghangli-uni]

This is from the pbs.error message

Loading access-om3/2025.08.000
  Loading requirement: openmpi/4.1.7-alvuihq access3-share/2025.08.000-4addtwr
    access-cice/CICE6.6.1-0-rmyd5d5 access-mom6/2025.07.000-zeetsan
    access-ww3/2025.08.000-7fuyev2 access3/2025.08.000-ujpt2mh
Loading openmpi/4.1.7-ushgfj4
  ERROR: Conflicting openmpi is loaded
Unloading openmpi/4.1.7-alvuihq
  ERROR: openmpi/4.1.7-alvuihq cannot be unloaded due to a prereq.
    HINT: Might try "module unload access3-share/2025.08.000-4addtwr
    access-cice/CICE6.6.1-0-rmyd5d5 access-mom6/2025.07.000-zeetsan
    access-ww3/2025.08.000-7fuyev2 access3/2025.08.000-ujpt2mh" first.
MODULE ERROR DETECTED: GLOBALERR Conflicting openmpi is loaded
(Detailed error information and backtrace has been suppressed, set $MODULES_ERROR_BACKTRACE to unsuppress.)

Loading openmpi/4.1.7
  ERROR: Conflicting openmpi is loaded
Currently Loaded Modulefiles:
 1) openmpi/4.1.7-alvuihq                       9) pbs
 2) access3-share/2025.08.000-4addtwr
 3) access-cice/CICE6.6.1-0-rmyd5d5
 4) access-mom6/2025.07.000-zeetsan
 5) access-ww3/2025.08.000-7fuyev2
 6) access3/2025.08.000-ujpt2mh
 7) access-om3/2025.08.000
 8) model-tools/mppnccombine-fast/2025.07.000
/g/data/tm70/ml0072/COMMON/git_repos/payu-org_payu/payu/runlog.py:108: UserWarning: Runlog commit was commited without git signing

Loading access-om3/2025.08.000 auto-loads openmpi/4.1.7-alvuihq. Then it tries to load a different openmpi (openmpi/4.1.7-ushgfj4 / generic openmpi/4.1.7), which conflicts.