PAYU issues on Leonardo

Natalia · 18 November 2024 01:51

Hello to everyone,

From what I found here, probably this question should be addressed to @Aidan @john_reilly @angus-g @dale.roberts @harshula

I have some progress with payu and porting ACCESS OM2 to Leonardo supercomputer.

Right now at the stage of payu run.

Long story short:

*.exe files are compiled with ACCESS NRI local modules (compiled with spack)
RYF JRA-55 files calculated locally on Leonardo
Initial conditions (transferred to Leonardo) and forcing fields specified in config.yaml and atmosphere/forcing.json

payu setup produced manifests, can be found in my local repo: GitHub - VanuatuN/1deg_jra55_ryf: 1 degree ACCESS-OM2 experiment with JRA55 RYF atmospheric forcing.

The questions are:

how to force payu to use my local modules that were compiled at the first stage?
I know it is going to be slower than with system modules, but I want to make it just
working first
where exactly in payu/*.py files I should modify the rest of the slurm specific flags for Leonardo?

Example of batch script:
#!/bin/bash
#SBATCH --job-name=benchmark_test
#SBATCH --output=benchmark_test.out
#SBATCH --error=benchmark_test.err
#SBATCH --nodes=1
#SBATCH --cpus-per-task=32
#SBATCH -A ICT24_MHPC
#SBATCH --time=00:30:00
#SBATCH --partition=boost_usr_prod

Thank you!

The current output from payu run:

02:48 $ payu run 
payu: warning: Job request includes 47 unused CPUs.
payu: warning: CPU request increased from 241 to 288
sbatch -A ICT24_MHPC --time=10800 --ntasks=288 --wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/
gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/
python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" --export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,MODULESHOME
=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/
environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=/leonardo/prod/spack/03/install/
0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,MODULEPATH=
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:/leonardo/prod/opt/modulefiles/
profiles:/leonardo/prod/opt/modulefiles/base/archive:/leonardo/prod/opt/modulefiles/
base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/
modulefiles/base/environment:/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/
prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:/leonardo/prod/opt/modulefiles/base/applications"
sbatch: error: no partition specified, using default partition lrd_all_serial
sbatch: error: no gres:tmpfs specified, using default: gres:tmpfs:10g
sbatch: error: Batch job submission failed: More processors requested than permitted
Traceback (most recent call last):
  File "/leonardo/home/userexternal/ntilinin/.local/bin/payu", line 10, in <module>
    sys.exit(parse())
             ^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/cli.py", line 42, in parse
    run_cmd(**args)
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/subcommands/run_cmd.py", line 108, in runcmd
    cli.submit_job('payu-run', pbs_config, pbs_vars)
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/cli.py", line 156, in submit_job
    subprocess.check_call(shlex.split(cmd))
  File "/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sbatch', '-A', 'ICT24_MHPC', '--time=10800', '--ntasks=288',
 '--wrap=/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/
anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/python 
/leonardo/home/userexternal/ntilinin/.local/bin/payu-run', '--export=PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/
bin,MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/
gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/
environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,
MODULEPATH=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:
/leonardo/prod/opt/modulefiles/profiles:/leonardo/prod/opt/modulefiles/
base/archive:/leonardo/prod/opt/modulefiles/base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/modulefiles/base/environment:
/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:/leonardo/prod/opt/modulefiles/base/applications']' returned non-zero exit status 1.

john_reilly · 18 November 2024 02:17

Hi Natalia,

I’m no expert on this stuff and just picked up where @angus-g and @ChrisC28 got to with our slurm-based hpc. But hopefully this helps;

Not sure about the first question sorry, but for the slurm specific flags, we put them at the start of the config.yaml file in the run directory. If you can find the slurm.py file in payu/schedulers/ that should make more sense about how payu reads these flags in.

Here’s an example of one of our config.yaml files:

scheduler: slurm
project: pawsey0410
walltime: 02:20:00
jobname: eac_sthpac-forced_v3
ncpus: 1804
nnodes: 15
runspersub: 1

shortpath: /scratch/pawsey0410
model: mom6
input:
    - /scratch/pawsey0410/jreilly/mom6-inputs/eac_sthpac-forced_v2/
    - /scratch/pawsey0410/jreilly/jra_padded/2016/
    - /scratch/pawsey0410/jreilly/mom6/archive/eac_sthpac-forced_v3/restart305
#    - /g/data/ua8/JRA55-do/RYF/v1-3/
#    - /g/data/ik11/inputs/JRA-55/RYF/v1-3/
# release exe
exe: /software/projects/pawsey0410/cc7576/mom6-cmake/coupler/MOM6-SIS2
  #exe: /software/projects/pawsey0410/jreilly/mom6-cmake/coupler/MOM6-SIS2
  #  /software/projects/pawsey0410/cc7576/mom6-cmake/coupler/MOM6-SIS2

stacksize: unlimited

collate: false
runlog: false

mpi:
  runcmd: srun

anton · 18 November 2024 02:53

Hi Natalia

This looks like where it failed:

It looks like the number of nodes is hardcoded to 1, and then payu is requesting 288 cores. I don’t know how many cores per node Leonardo hardware has, for us its 48, so setting the number of nodes to 6 would be correct for us. I would try setting ncpus and nnodes in the config.yaml per Johns code snippet…

There’s these two lines in the payu output:

payu: warning: Job request includes 47 unused CPUs.
payu: warning: CPU request increased from 241 to 288

I think there might be 32 cores per node for you, so I would try:

ncpus: 256
nnodes: 8
npernode: 32

For our normal gadi scheduler, we don’t specify the number of nodes, so its possible Payu hasn’t been tested very well in these cases.

Re: modules

You can set it similar to this config:

github.com

ACCESS-NRI/access-om3-configs/blob/01cbd6fceb970ab79ee82490df3d4bf8d41447e8/config.yaml#L47-L51


      
          use:
              - /g/data/vk83/modules
          load:
              - access-om3/2024.09.0
              - nco/5.0.5

When you have module: use: and load: lines in the config.yaml, you should be able to access the binaries without a path

for example, the exe: entry just could become yatm.exe

I would test these in a command prompt first by doing a module use and module load, and seeing if the executables are available as commands.

Natalia · 19 November 2024 00:01

Hi @john_reilly, very much appreciated!
Didn’t know that slurm flags can be specified exactly in config.yaml

Will try to implement it.

Natalia · 19 November 2024 00:03

Hi @anton!

Thank you, all clear for the moment!
My time zone forces for a delay in reply.

Very useful information, will do that and let you know soon.

Fingers crossed.

anton · 19 November 2024 00:26

Happy to help. If the instructions don’t make sense , I can make a Pull Request into your fork of the om2 configurations

Natalia · 19 November 2024 20:34

It worked for the job submission! But failed to pick the modules.
Leonardo has 32 cores per node, right.
I modified slurm.py and config.yaml files

payu run gives:

payu run 
sbatch -A ICT24_MHPC --time=00:30:00 --ntasks=256 --partition=boost_usr_prod 
--wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/
anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/
python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" --export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,
MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/
gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=/leonardo/prod/spack/03/
install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,MODULEPATH=
/leonardo_scratch/large/userexternal/ntilinin/ACCESS-NRI/release/
modules/linux-rhel8-x86_64:/leonardo/prod/spack/03/install/0.19/
linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:/leonardo/prod/opt/modulefiles/
profiles:/leonardo/prod/opt/modulefiles/base/archive:/leonardo/prod/opt/modulefiles/
base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/
modulefiles/base/environment:/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/
prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:
/leonardo/prod/opt/modulefiles/base/applications"

But still an output from payu run is:

laboratory path:  ./ntilinin/access-om2
binary path:  ./ntilinin/access-om2/bin
input path:  ./ntilinin/access-om2/input
work path:  ./ntilinin/access-om2/work
archive path:  ./ntilinin/access-om2/archive
nruns: 1 nruns_per_submit: 1 subrun: 1
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up access-om2
Checking exe and input manifests
Updating full hashes for 3 files in manifests/exe.yaml
Creating restart manifest
Writing manifests/restart.yaml
Writing manifests/exe.yaml
payu: Found modules in /leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi
Traceback (most recent call last):
  File "/leonardo/home/userexternal/ntilinin/.local/bin/payu-run", line 10, in <module>
    sys.exit(runscript())
             ^^^^^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/subcommands/run_cmd.py", line 132, in runscript
    expt.run()
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/experiment.py", line 517, in run
    mpi_module = envmod.lib_update(
                 ^^^^^^^^^^^^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/envmod.py", line 114, in lib_update
    mod_name, mod_version = fsops.splitpath(lib_path)[2:4]
    ^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 0)

modified slurm.py file:

github.com

VanuatuN/payu/blob/7372eb99e1a1b04e405203e5c053b5a12c87d357/payu/schedulers/slurm.py#L40-L50


      
            # Retrieve project and append to flags
            pbs_project = pbs_config.get('project', os.environ.get('PROJECT', 'ICT24_MHPC'))
            pbs_flags.append('-A {project}'.format(project=pbs_project))
          
           # Retrieve walltime and ntasks and append to flags
            pbs_flags.append('--time={}'.format(pbs_config.get('walltime'))) 
            pbs_flags.append('--ntasks={}'.format(pbs_config.get('ncpus')))
          
          # Add partition flag
            pbs_partition = pbs_config.get('partition', os.environ.get('PARTITION', 'boost_usr_prod'))
            pbs_flags.append('--partition={}'.format(pbs_partition))

modified (with nodes, etc. specified)config.yaml file:

github.com

VanuatuN/1deg_jra55_ryf/blob/34d8b24129c143043e3613a61ce25cfae5b00522/config.yaml#L10-L20


      
          
          scheduler: slurm
          queue: boost_usr_prod
          walltime: 00:30:00
          jobname: 1deg_jra55_ryf
          project: ICT24_MHPC
          jobname: access_om2_ryf
          ncpus: 256
          nnodes: 8
          npernode: 32
          #mem: 1000GB

Will try to work around with modules in the coming days. But
would very grateful for any hints where to move.

Thank you!!!

anton · 19 November 2024 21:53

It looks like payu is trying to check that the mpi version which is linked to by the model executable its the version loaded. But for whatever reason the formatting or check is failing.

I would try adding these lines to your config.yaml and set them to the modules which are used by your exectuables. (You might be able to confirm the path to the mpi version using ldd )

mpi:
    modulepath:
    module:

See this section in the docs:

https://payu.readthedocs.io/en/stable/config.html#miscellaneous

Pinging @Aidan as he has more experience than I with this!

Aidan · 19 November 2024 22:44

Yes this was always quite NCI specific, and with the spack built executables is no longer strictly necessary.

Can you try updating your version of payu, as there is now logic that isolates this check to NCI systems by matching the library path:

github.com

payu-org/payu/blob/master/payu/envmod.py#L118


      
              exec(envs)
          
          
          def lib_update(required_libs, lib_name):
              # Local import to avoid reversion interference
              # TODO: Bad design, fixme!
              # NOTE: We may be able to move this now that reversion is going away
              from payu import fsops
          
              for lib_filename, lib_path in required_libs.items():
                  if lib_filename.startswith(lib_name) and lib_path.startswith('/apps/'): 
                      # Load nci's /apps/ version of module if required 
                      # pylint: disable=unbalanced-tuple-unpacking
                      mod_name, mod_version = fsops.splitpath(lib_path)[2:4]
          
                      module('unload', mod_name)
                      module('load', os.path.join(mod_name, mod_version))
                      return '{0}/{1}'.format(mod_name, mod_version)
          
              # If there are no libraries, return an empty string
              return ''

If you have made local changes you can fetch the latest payu and git rebase your changes on top of them.

Natalia · 24 November 2024 18:30

Update:

Before I was using ‘pawsey’ branch from the payu repo (found somewhere is the issues or here on forum). Now switched to ‘master’ branch and made few corrections.

The job goes to submission, which is goods news.

My opempi module does not recognise --chdir, so I commented out this string and kept -wdir:

github.com

VanuatuN/payu/blob/aa1f7edc01ff94dee23575d61e5837459c2f61b8/payu/experiment.py#L582-L586


      
          wdir_arg = '-wdir'
          #if self.config.get('scheduler') == 'slurm':
          #    # Option to set the working directory differs in slurm
          #    wdir_arg = '--chdir'
          model_prog.append(f'{wdir_arg} {model.work_path}')

What I’m not sure about is that payu uses the correct version of as the cmd still looks as:

 ~/access-om2/control/1deg_jra55_ryf [master ↑·13|…28] 
15:57 $ payu run 
/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/fsops.py:77: UserWarning: Duplicate key found in config.yaml: key 'jobname' with value 'access_om2_ryf'. This overwrites the original value: '1deg_jra55_ryf'
/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/fsops.py:77: UserWarning: Duplicate key found in config.yaml: key 'queue' with value 'boost_usr_prod'. This overwrites the original value: 'boost_usr_prod'
sbatch -A ICT24_MHPC --time=00:30:00 --ntasks=256 --partition=boost_usr_prod 
--wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/
gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/
bin/python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" 
--export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,
MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,
MODULES_CMD=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,
MODULEPATH=/leonardo_scratch/large/userexternal/ntilinin/ACCESS-NRI/release/modules/linux-rhel8-x86_64:
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:
/leonardo/prod/opt/
modulefiles/profiles:
/leonardo/prod/opt/modulefiles/base/archive:
/leonardo/prod/opt/modulefiles/base/dependencies:
/leonardo/prod/opt/modulefiles/base/data:
/leonardo/prod/opt/modulefiles/base/environment:
/leonardo/prod/opt/modulefiles/base/libraries:
/leonardo/prod/opt/modulefiles/base/tools:
/leonardo/prod/opt/modulefiles/base/compilers:
/leonardo/prod/opt/modulefiles/base/applications"
Submitted batch job 9594255

It still sees MODULES_CMD and use systemwide modulecmd.tcl as well as systemwide MODULESHOME, not sure it affects something, but still.

However MODULEPATH is updated to the proper location.

Slurm out looks reasonable, hope it picks not default systemwide openmpi, but first in the list:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/slurm-9597369.out#L20-L49


      
          Checking exe, input and restart manifests
          Unloading profile/base
            ERROR: Module evaluation aborted
          ERROR: Unable to locate a modulefile for 'slurm'
          Loading openmpi/4.1.6--gcc--12.2.0
            Loading requirement: gcc/12.2.0
          Currently Loaded Modulefiles:
           1) profile/base(default)                               
           2) oasis3-mct/master <aL>                              
           3) libaccessom2/master <aL>                            
           4) cice5/master <aL>                                   
           5) mom5/master <aL>                                    
           6) access-om2/latest                                   
           7) linux-rhel8-x86_64/intel-oneapi-compilers/2021.2.0  
           8) gcc/12.2.0(default) <aL>                            
           9) openmpi/4.1.6--gcc--12.2.0(default)                 
          
          Key:
          (symbolic-version)  <module-tag>  <aL>=auto-loaded  
          /leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/runlog.py:110: UserWarning: Error occured when attempting to commit runlog

This file has been truncated. show original

Another issue: I’m doing something wrong with resources allocation,
don’t know how payu distributes submodels across nodes, here is an error that I’m getting now:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/access-om2.err

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 16
slots that were requested by the application:

  ./ICT24_MHPC/ntilinin/access-om2/work/1deg_jra55_ryf-438f4db9/ocean/fms_ACCESS-OM_1883a84_libaccessom2_7ce7920.x

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

This file has been truncated. show original

And the config.yaml:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/config.yaml#L51-L60


      
                - /leonardo_scratch/fast/ICT24_MHPC/ntilinin/INPUT/access-om2/ocean/processor_masks/global.1deg/216.16x15/2020.05.30
                - /leonardo_scratch/fast/ICT24_MHPC/ntilinin/INPUT/access-om2/ocean/tides/global.1deg/2020.05.30
          scheduler: slurm
          project: ICT24_MHPC
          walltime: 00:30:00
          queue: boost_usr_prod
          ncpus: 256
          nnodes: 8
          npernode: 32

I’ll try to work around, but would be grateful for any advice as usual

Many thanks!!!

anton · 24 November 2024 21:53

Hi Natalia

I would try commenting out these lines:

github.com

payu-org/payu/blob/27aac372dba2752e1d58ab1d66db799e7891e103/payu/experiment.py#L606-L608


      
          if model_npernode % 2 == 0:
              npernode_flag = ('-map-by ppr:{0}:socket'
                               ''.format(model_npernode / 2))

For reasons that are not clear to me, for some reason payu is requesting 16 tasks per available “socket”, when its probably only possible to have one. Maybe this is a gadi specific detail for some specific case. You might be able to remove the -map-by argument entirely. I think it will take some experimentation.

Natalia · 24 November 2024 22:09

Hi Anton,

Will try, thank you.
It could be that each gadi’s node has 3 sockets each with 16 cores, this makes sense - 16*3=48 cores.

Natalia · 14 February 2025 15:12

Hi @anton

I have an update and a question again.

So far I’ve been struggling during the last month to make payu run executables built from old COSIMA repo with spack built modules (from ACCESS NRI). I rewrote parts of payu that checks modules and adds libraries, but couldn’t make it use proper mpi. Gave up that.

Now I cloned the latest version of payu and trying to run spack built executables with spack built model components from cloned config repo GitHub - ACCESS-NRI/access-om2-configs at release-1deg_jra55_ryf. The problem I’m facing now is that one:

mpirun was unable to launch the specified application as it could not access
or execute an executable:

#--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not access
or execute an executable:

Executable: ./ntilinin/access-om2/work/1deg_jra55_ryf-expt-3216c7cb/atmosphere/yatm.exe
Node: lrdn3421

while attempting to start process rank 0.
#--------------------------------------------------------------------------

“which mpirun” points to the proper module prebuilt with spack.
The symlink points to the proper location of yamt.exe

I do have a feeling that it again uses systemwide mpirun (just a guess).

I found the same issue raised by @Aidan here ACCESS-OM2 Restart Reproducibility: Bitwise Reproducibility Testing

Maybe there is something specific you and @Aidan can advise on that issues?

Many thanks as usual!

Also an update on nodes/sockets/cores:

It should be 2 sockets per node on Gadi

And it is 1 socket with 32 cores on each Leonardo node, so no need to divide by 2. Payu was configured to divide 32/2 in case of even number ‘npernode’ from config.yaml

Natalia · 15 February 2025 10:49

This is solved now.

The problem was these parts of the code:

github.com/payu-org/payu

payu/experiment.py

d0fdbff1c


      
          wdir_arg = '-wdir'
          if self.config.get('scheduler') == 'slurm':
              # Option to set the working directory differs in slurm
              wdir_arg = '--chdir'
          model_prog.append(f'{wdir_arg} {model.work_path}')

And here:

github.com/payu-org/payu

payu/experiment.py

d0fdbff1c


      
          # Use the full path to symlinked exec_name in work as some
          # older MPI libraries complained executable was not in PATH
          model_prog.append(os.path.join(model.work_path, model.exec_name))

When using -wdir argument with slurm and mpi (probably Leonardo version of slurm, but more likely in general) the line resulting from run_cmd looks as:

mpirun --mca io ompio --mca io_ompio_num_aggregators 1 -wdir ./ntilinin/access-om2/work/1deg_jra55_ryf-expt-a3822e12/atmosphere -n 1 ./ntilinin/access-om2/work/1deg_jra55_ryf-expt-a3822e12/atmosphere/yatm.exe

and after working directory has been changed to:

/ntilinin/access-om2/work/1deg_jra55_ryf-expt-a3822e12/atmosphere

it looks for the path to *.exe files starting from there. So the relative path won’t work.
I’ve changed the line 629 in experiment.py to

model_prog.append(os.path.abspath(os.path.join(model.work_path, model.exec_name)))

And mpirun picked to file.

Now I’m facing that output in the access-om2.err:

--------------------------------------------------------------------------
ORTE has lost communication with a remote daemon.

  HNP daemon   : [[55189,0],0] on node lrdn0001
  Remote daemon: [[55189,0],1] on node lrdn0009

This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
fms_ACCESS-OM.x    0000000001D5078B  Unknown               Unknown  Unknown
libpthread-2.28.s  000014F4C28ECCF0  Unknown               Unknown  Unknown
libopen-rte.so.40  000014F4BE89CD30  orte_dt_init          Unknown  Unknown
libopen-rte.so.40  000014F4BE8E4BB9  orte_ess_base_std     Unknown  Unknown
libopen-rte.so.40  000014F4BE8E8AA2  Unknown               Unknown  Unknown
libopen-rte.so.40  000014F4BE95CF0B  orte_init             Unknown  Unknown
libmpi.so.40.30.4  000014F4C3101CE1  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.30.4  000014F4C2F2F40D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  000014F4C3440787  PMPI_Init_f08         Unknown  Unknown
fms_ACCESS-OM.x    0000000001D23B41  coupler_mod_mp_co          82  coupler.F90
fms_ACCESS-OM.x    000000000041F8A0  MAIN__                    186  ocean_solo.F90
fms_ACCESS-OM.x    00000000004111E2  Unknown               Unknown  Unknown
libc-2.28.so       000014F4C254ED85  __libc_start_main     Unknown  Unknown
fms_ACCESS-OM.x    00000000004110EE  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)

There is still a manual memory allocation in run_cmd.py in the lines 36-39, that I changed to Leonardo specs:

# TODO: Create drivers for servers
platform = pbs_config.get('platform', {})
max_cpus_per_node = platform.get('nodesize', 32)
max_ram_per_node = platform.get('nodemem', 514)

Probably it gets overwritten from config.yaml, but still.
Don’t know here to dig now for ORTE error.

Would be grateful for any guess as usual!

anton · 17 February 2025 00:33

Glad to hear you are making some progress!

Not sure I have much to offer.

Yes - this is strange. Can you confirm which code versions you are using ?

It looks like its failing at MPI_Init for MOM

github.com/ACCESS-NRI/libaccessom2

libcouple/src/coupler.F90

f9f2d67a4


      
          ! Validate model names
          call assert(model_name == 'matmxx' .or. model_name == 'cicexx' &
                      .or. model_name == 'mom5xx' .or. model_name == 'monito', &
                      'Bad model name')
          
          self%num_fields = 0
          self%model_name = model_name
          
          call MPI_Initialized(initialized, err)
          if (.not. initialized) then
              call MPI_Init(err)
          endif
          
          if (present(loggerin)) then
              self%logger => loggerin
          else
              self%logger => null()
          endif
          
          if (present(config_dir)) then
              call oasis_init_comp(self%comp_id, model_name, err, config_dir=config_dir)

I guess the first thing to confirm is that the mpirun command printed from run_cmd has the right number of processors in the -n argument for MOM? The MOM exectuable is named fms_ACCESS-OM.x (And are the number of processors consistent with config.yaml)

angus-g · 17 February 2025 03:19

I don’t know the specifics of Leonardo, but I did think that SLURM-based systems had to use srun as a “wrapper” for MPI (it actually does a lot about setting up the execution environment), rather than trying to call mpiexec directly. That’s what we had to do on Setonix, but I’m not sure if that continues to be the approach? Apologies if this is a red herring!

Aidan · 17 February 2025 05:34

I’d definitely be asking your local HPC Helpdesk @Natalia to see if they can assist with this error. I am sure they’d have seen similar issues and be able to advise.

Please also note we’re doing some exploratory work to port ACCESS models to Setonix, a SLURM based machine. I can’t give you a definite time-line, but I would think we’d have made some progress within a month.

Natalia · 17 February 2025 20:41

Not a red herring! Makes sense, tried it and got:

The application appears to have been direct launched using “srun”,
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren’t able
to discover a SLURM installation in the usual places.

Please configure as appropriate and try again.

The thing is that I’m using a spack built OMPI from ACCESS NRI repo,
it was likely built without SLURM support.

which srun points to systemwide executable even the ACCESS NRI module is loaded:

21:23 $ which srun 
/usr/bin/srun

Also srun do not support flags like `–mca io ompio --mca io_ompio_num_aggregators 1, not sure whether they needed.

From what I read here 10.7. Launching with Slurm — Open MPI main documentation
mpirun is the recommended method for launching Open MPI jobs in Slurm jobs (at least now I know ).
Thanks!

anton · 17 February 2025 21:47

My guess is that these settings are not essential for the model to run, and that the defaults probably work.

The thing is that I’m using a spack built OMPI from ACCESS NRI repo,
it was likely built without SLURM support.

It’s possibly worth revisiting this and using the system provided OpenMPI. On gadi, we found the system provided version performed better than the version we built through spack. If there is someone at your local helpdesk who understands spack, they may have advice ?

Topic		Replies	Views
Perturbation experiments forked from ACCESS-OM2-025 RYF control run on ik11 Ocean help , access-om2 , solved	20	108	5 November 2024
PAYU issues on Setonix Technical payu	17	424	26 September 2023
Running ACCESS-OM2 on Leonardo supercomputer Technical help , access-om2 , inscope	42	553	17 November 2024
21st March 2025 - How to run an ACCESS-OM2 or OM3 model 2025 training program cosima , access-om2 , payu , training	0	56	20 February 2025
Development in payu General help	4	31	17 February 2025

Related topics