PAYU issues on Leonardo

Hi @anton

I have an update and a question again.

So far I’ve been struggling during the last month to make payu run executables built from old COSIMA repo with spack built modules (from ACCESS NRI). I rewrote parts of payu that checks modules and adds libraries, but couldn’t make it use proper mpi. Gave up that.

Now I cloned the latest version of payu and trying to run spack built executables with spack built model components from cloned config repo GitHub - ACCESS-NRI/access-om2-configs at release-1deg_jra55_ryf. The problem I’m facing now is that one:

mpirun was unable to launch the specified application as it could not access
or execute an executable:

#--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not access
or execute an executable:

Executable: ./ntilinin/access-om2/work/1deg_jra55_ryf-expt-3216c7cb/atmosphere/yatm.exe
Node: lrdn3421

while attempting to start process rank 0.
#--------------------------------------------------------------------------

“which mpirun” points to the proper module prebuilt with spack.
The symlink points to the proper location of yamt.exe

I do have a feeling that it again uses systemwide mpirun (just a guess).

I found the same issue raised by @Aidan here ACCESS-OM2 Restart Reproducibility: Bitwise Reproducibility Testing

Maybe there is something specific you and @Aidan can advise on that issues?

Many thanks as usual!

Also an update on nodes/sockets/cores:

It should be 2 sockets per node on Gadi

And it is 1 socket with 32 cores on each Leonardo node, so no need to divide by 2. Payu was configured to divide 32/2 in case of even number ‘npernode’ from config.yaml