PAYU issues on Leonardo

Natalia · 18 November 2024 01:51

Hello to everyone,

From what I found here, probably this question should be addressed to @Aidan @john_reilly @angus-g @dale.roberts @harshula

I have some progress with payu and porting ACCESS OM2 to Leonardo supercomputer.

Right now at the stage of payu run.

Long story short:

*.exe files are compiled with ACCESS NRI local modules (compiled with spack)
RYF JRA-55 files calculated locally on Leonardo
Initial conditions (transferred to Leonardo) and forcing fields specified in config.yaml and atmosphere/forcing.json

payu setup produced manifests, can be found in my local repo: GitHub - VanuatuN/1deg_jra55_ryf: 1 degree ACCESS-OM2 experiment with JRA55 RYF atmospheric forcing.

The questions are:

how to force payu to use my local modules that were compiled at the first stage?
I know it is going to be slower than with system modules, but I want to make it just
working first
where exactly in payu/*.py files I should modify the rest of the slurm specific flags for Leonardo?

Example of batch script:
#!/bin/bash
#SBATCH --job-name=benchmark_test
#SBATCH --output=benchmark_test.out
#SBATCH --error=benchmark_test.err
#SBATCH --nodes=1
#SBATCH --cpus-per-task=32
#SBATCH -A ICT24_MHPC
#SBATCH --time=00:30:00
#SBATCH --partition=boost_usr_prod

Thank you!

The current output from payu run:

02:48 $ payu run 
payu: warning: Job request includes 47 unused CPUs.
payu: warning: CPU request increased from 241 to 288
sbatch -A ICT24_MHPC --time=10800 --ntasks=288 --wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/
gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/
python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" --export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,MODULESHOME
=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/
environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=/leonardo/prod/spack/03/install/
0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,MODULEPATH=
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:/leonardo/prod/opt/modulefiles/
profiles:/leonardo/prod/opt/modulefiles/base/archive:/leonardo/prod/opt/modulefiles/
base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/
modulefiles/base/environment:/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/
prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:/leonardo/prod/opt/modulefiles/base/applications"
sbatch: error: no partition specified, using default partition lrd_all_serial
sbatch: error: no gres:tmpfs specified, using default: gres:tmpfs:10g
sbatch: error: Batch job submission failed: More processors requested than permitted
Traceback (most recent call last):
  File "/leonardo/home/userexternal/ntilinin/.local/bin/payu", line 10, in <module>
    sys.exit(parse())
             ^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/cli.py", line 42, in parse
    run_cmd(**args)
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/subcommands/run_cmd.py", line 108, in runcmd
    cli.submit_job('payu-run', pbs_config, pbs_vars)
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/cli.py", line 156, in submit_job
    subprocess.check_call(shlex.split(cmd))
  File "/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sbatch', '-A', 'ICT24_MHPC', '--time=10800', '--ntasks=288',
 '--wrap=/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/
anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/python 
/leonardo/home/userexternal/ntilinin/.local/bin/payu-run', '--export=PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/
bin,MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/
gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/
environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,
MODULEPATH=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:
/leonardo/prod/opt/modulefiles/profiles:/leonardo/prod/opt/modulefiles/
base/archive:/leonardo/prod/opt/modulefiles/base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/modulefiles/base/environment:
/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:/leonardo/prod/opt/modulefiles/base/applications']' returned non-zero exit status 1.

john_reilly · 18 November 2024 02:17

Hi Natalia,

I’m no expert on this stuff and just picked up where @angus-g and @ChrisC28 got to with our slurm-based hpc. But hopefully this helps;

Not sure about the first question sorry, but for the slurm specific flags, we put them at the start of the config.yaml file in the run directory. If you can find the slurm.py file in payu/schedulers/ that should make more sense about how payu reads these flags in.

Here’s an example of one of our config.yaml files:

scheduler: slurm
project: pawsey0410
walltime: 02:20:00
jobname: eac_sthpac-forced_v3
ncpus: 1804
nnodes: 15
runspersub: 1

shortpath: /scratch/pawsey0410
model: mom6
input:
    - /scratch/pawsey0410/jreilly/mom6-inputs/eac_sthpac-forced_v2/
    - /scratch/pawsey0410/jreilly/jra_padded/2016/
    - /scratch/pawsey0410/jreilly/mom6/archive/eac_sthpac-forced_v3/restart305
#    - /g/data/ua8/JRA55-do/RYF/v1-3/
#    - /g/data/ik11/inputs/JRA-55/RYF/v1-3/
# release exe
exe: /software/projects/pawsey0410/cc7576/mom6-cmake/coupler/MOM6-SIS2
  #exe: /software/projects/pawsey0410/jreilly/mom6-cmake/coupler/MOM6-SIS2
  #  /software/projects/pawsey0410/cc7576/mom6-cmake/coupler/MOM6-SIS2

stacksize: unlimited

collate: false
runlog: false

mpi:
  runcmd: srun

anton · 18 November 2024 02:53

Hi Natalia

This looks like where it failed:

It looks like the number of nodes is hardcoded to 1, and then payu is requesting 288 cores. I don’t know how many cores per node Leonardo hardware has, for us its 48, so setting the number of nodes to 6 would be correct for us. I would try setting ncpus and nnodes in the config.yaml per Johns code snippet…

There’s these two lines in the payu output:

payu: warning: Job request includes 47 unused CPUs.
payu: warning: CPU request increased from 241 to 288

I think there might be 32 cores per node for you, so I would try:

ncpus: 256
nnodes: 8
npernode: 32

For our normal gadi scheduler, we don’t specify the number of nodes, so its possible Payu hasn’t been tested very well in these cases.

Re: modules

You can set it similar to this config:

github.com

ACCESS-NRI/access-om3-configs/blob/01cbd6fceb970ab79ee82490df3d4bf8d41447e8/config.yaml#L47-L51


      
          use:
              - /g/data/vk83/modules
          load:
              - access-om3/2024.09.0
              - nco/5.0.5

When you have module: use: and load: lines in the config.yaml, you should be able to access the binaries without a path

for example, the exe: entry just could become yatm.exe

I would test these in a command prompt first by doing a module use and module load, and seeing if the executables are available as commands.

Natalia · 19 November 2024 00:01

Hi @john_reilly, very much appreciated!
Didn’t know that slurm flags can be specified exactly in config.yaml

Will try to implement it.

Natalia · 19 November 2024 00:03

Hi @anton!

Thank you, all clear for the moment!
My time zone forces for a delay in reply.

Very useful information, will do that and let you know soon.

Fingers crossed.

anton · 19 November 2024 00:26

Happy to help. If the instructions don’t make sense , I can make a Pull Request into your fork of the om2 configurations

Natalia · 19 November 2024 20:34

It worked for the job submission! But failed to pick the modules.
Leonardo has 32 cores per node, right.
I modified slurm.py and config.yaml files

payu run gives:

payu run 
sbatch -A ICT24_MHPC --time=00:30:00 --ntasks=256 --partition=boost_usr_prod 
--wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/
anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/bin/
python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" --export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,
MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/
gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,MODULES_CMD=/leonardo/prod/spack/03/
install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,MODULEPATH=
/leonardo_scratch/large/userexternal/ntilinin/ACCESS-NRI/release/
modules/linux-rhel8-x86_64:/leonardo/prod/spack/03/install/0.19/
linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:/leonardo/prod/opt/modulefiles/
profiles:/leonardo/prod/opt/modulefiles/base/archive:/leonardo/prod/opt/modulefiles/
base/dependencies:/leonardo/prod/opt/modulefiles/base/data:/leonardo/prod/opt/
modulefiles/base/environment:/leonardo/prod/opt/modulefiles/base/libraries:/leonardo/
prod/opt/modulefiles/base/tools:/leonardo/prod/opt/modulefiles/base/compilers:
/leonardo/prod/opt/modulefiles/base/applications"

But still an output from payu run is:

laboratory path:  ./ntilinin/access-om2
binary path:  ./ntilinin/access-om2/bin
input path:  ./ntilinin/access-om2/input
work path:  ./ntilinin/access-om2/work
archive path:  ./ntilinin/access-om2/archive
nruns: 1 nruns_per_submit: 1 subrun: 1
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up access-om2
Checking exe and input manifests
Updating full hashes for 3 files in manifests/exe.yaml
Creating restart manifest
Writing manifests/restart.yaml
Writing manifests/exe.yaml
payu: Found modules in /leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi
Traceback (most recent call last):
  File "/leonardo/home/userexternal/ntilinin/.local/bin/payu-run", line 10, in <module>
    sys.exit(runscript())
             ^^^^^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/subcommands/run_cmd.py", line 132, in runscript
    expt.run()
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/experiment.py", line 517, in run
    mpi_module = envmod.lib_update(
                 ^^^^^^^^^^^^^^^^^^
  File "/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/envmod.py", line 114, in lib_update
    mod_name, mod_version = fsops.splitpath(lib_path)[2:4]
    ^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 0)

modified slurm.py file:

github.com

VanuatuN/payu/blob/7372eb99e1a1b04e405203e5c053b5a12c87d357/payu/schedulers/slurm.py#L40-L50


      
            # Retrieve project and append to flags
            pbs_project = pbs_config.get('project', os.environ.get('PROJECT', 'ICT24_MHPC'))
            pbs_flags.append('-A {project}'.format(project=pbs_project))
          
           # Retrieve walltime and ntasks and append to flags
            pbs_flags.append('--time={}'.format(pbs_config.get('walltime'))) 
            pbs_flags.append('--ntasks={}'.format(pbs_config.get('ncpus')))
          
          # Add partition flag
            pbs_partition = pbs_config.get('partition', os.environ.get('PARTITION', 'boost_usr_prod'))
            pbs_flags.append('--partition={}'.format(pbs_partition))

modified (with nodes, etc. specified)config.yaml file:

github.com

VanuatuN/1deg_jra55_ryf/blob/34d8b24129c143043e3613a61ce25cfae5b00522/config.yaml#L10-L20


      
          
          scheduler: slurm
          queue: boost_usr_prod
          walltime: 00:30:00
          jobname: 1deg_jra55_ryf
          project: ICT24_MHPC
          jobname: access_om2_ryf
          ncpus: 256
          nnodes: 8
          npernode: 32
          #mem: 1000GB

Will try to work around with modules in the coming days. But
would very grateful for any hints where to move.

Thank you!!!

anton · 19 November 2024 21:53

It looks like payu is trying to check that the mpi version which is linked to by the model executable its the version loaded. But for whatever reason the formatting or check is failing.

I would try adding these lines to your config.yaml and set them to the modules which are used by your exectuables. (You might be able to confirm the path to the mpi version using ldd )

mpi:
    modulepath:
    module:

See this section in the docs:

https://payu.readthedocs.io/en/stable/config.html#miscellaneous

Pinging @Aidan as he has more experience than I with this!

Aidan · 19 November 2024 22:44

Yes this was always quite NCI specific, and with the spack built executables is no longer strictly necessary.

Can you try updating your version of payu, as there is now logic that isolates this check to NCI systems by matching the library path:

github.com

payu-org/payu/blob/master/payu/envmod.py#L118


      
              exec(envs)
          
          
          def lib_update(required_libs, lib_name):
              # Local import to avoid reversion interference
              # TODO: Bad design, fixme!
              # NOTE: We may be able to move this now that reversion is going away
              from payu import fsops
          
              for lib_filename, lib_path in required_libs.items():
                  if lib_filename.startswith(lib_name) and lib_path.startswith('/apps/'): 
                      # Load nci's /apps/ version of module if required 
                      # pylint: disable=unbalanced-tuple-unpacking
                      mod_name, mod_version = fsops.splitpath(lib_path)[2:4]
          
                      module('unload', mod_name)
                      module('load', os.path.join(mod_name, mod_version))
                      return '{0}/{1}'.format(mod_name, mod_version)
          
              # If there are no libraries, return an empty string
              return ''

If you have made local changes you can fetch the latest payu and git rebase your changes on top of them.

Natalia · 24 November 2024 18:30

Update:

Before I was using ‘pawsey’ branch from the payu repo (found somewhere is the issues or here on forum). Now switched to ‘master’ branch and made few corrections.

The job goes to submission, which is goods news.

My opempi module does not recognise --chdir, so I commented out this string and kept -wdir:

github.com

VanuatuN/payu/blob/aa1f7edc01ff94dee23575d61e5837459c2f61b8/payu/experiment.py#L582-L586


      
          wdir_arg = '-wdir'
          #if self.config.get('scheduler') == 'slurm':
          #    # Option to set the working directory differs in slurm
          #    wdir_arg = '--chdir'
          model_prog.append(f'{wdir_arg} {model.work_path}')

What I’m not sure about is that payu uses the correct version of as the cmd still looks as:

 ~/access-om2/control/1deg_jra55_ryf [master ↑·13|…28] 
15:57 $ payu run 
/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/fsops.py:77: UserWarning: Duplicate key found in config.yaml: key 'jobname' with value 'access_om2_ryf'. This overwrites the original value: '1deg_jra55_ryf'
/leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/fsops.py:77: UserWarning: Duplicate key found in config.yaml: key 'queue' with value 'boost_usr_prod'. This overwrites the original value: 'boost_usr_prod'
sbatch -A ICT24_MHPC --time=00:30:00 --ntasks=256 --partition=boost_usr_prod 
--wrap="/leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/
gcc-8.5.0/anaconda3-2023.09-0-zcre7pfofz45c3btxpdk5zvcicdq5evx/
bin/python /leonardo/home/userexternal/ntilinin/.local/bin/payu-run" 
--export="PAYU_PATH=/leonardo/home/userexternal/ntilinin/.local/bin,
MODULESHOME=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi,
MODULES_CMD=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/libexec/modulecmd.tcl,
MODULEPATH=/leonardo_scratch/large/userexternal/ntilinin/ACCESS-NRI/release/modules/linux-rhel8-x86_64:
/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-8.5.0/environment-modules-5.2.0-rz47odw4phlhzhhbz7b65nv5s5othgmi/modulefiles:
/leonardo/prod/opt/
modulefiles/profiles:
/leonardo/prod/opt/modulefiles/base/archive:
/leonardo/prod/opt/modulefiles/base/dependencies:
/leonardo/prod/opt/modulefiles/base/data:
/leonardo/prod/opt/modulefiles/base/environment:
/leonardo/prod/opt/modulefiles/base/libraries:
/leonardo/prod/opt/modulefiles/base/tools:
/leonardo/prod/opt/modulefiles/base/compilers:
/leonardo/prod/opt/modulefiles/base/applications"
Submitted batch job 9594255

It still sees MODULES_CMD and use systemwide modulecmd.tcl as well as systemwide MODULESHOME, not sure it affects something, but still.

However MODULEPATH is updated to the proper location.

Slurm out looks reasonable, hope it picks not default systemwide openmpi, but first in the list:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/slurm-9597369.out#L20-L49


      
          Checking exe, input and restart manifests
          Unloading profile/base
            ERROR: Module evaluation aborted
          ERROR: Unable to locate a modulefile for 'slurm'
          Loading openmpi/4.1.6--gcc--12.2.0
            Loading requirement: gcc/12.2.0
          Currently Loaded Modulefiles:
           1) profile/base(default)                               
           2) oasis3-mct/master <aL>                              
           3) libaccessom2/master <aL>                            
           4) cice5/master <aL>                                   
           5) mom5/master <aL>                                    
           6) access-om2/latest                                   
           7) linux-rhel8-x86_64/intel-oneapi-compilers/2021.2.0  
           8) gcc/12.2.0(default) <aL>                            
           9) openmpi/4.1.6--gcc--12.2.0(default)                 
          
          Key:
          (symbolic-version)  <module-tag>  <aL>=auto-loaded  
          /leonardo/home/userexternal/ntilinin/.local/lib/python3.11/site-packages/payu/runlog.py:110: UserWarning: Error occured when attempting to commit runlog

This file has been truncated. show original

Another issue: I’m doing something wrong with resources allocation,
don’t know how payu distributes submodels across nodes, here is an error that I’m getting now:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/access-om2.err

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 16
slots that were requested by the application:

  ./ICT24_MHPC/ntilinin/access-om2/work/1deg_jra55_ryf-438f4db9/ocean/fms_ACCESS-OM_1883a84_libaccessom2_7ce7920.x

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

This file has been truncated. show original

And the config.yaml:

github.com

VanuatuN/1deg_jra55_ryf/blob/a4b5e55151eb1fceec779caa19e25230453be8ee/config.yaml#L51-L60


      
                - /leonardo_scratch/fast/ICT24_MHPC/ntilinin/INPUT/access-om2/ocean/processor_masks/global.1deg/216.16x15/2020.05.30
                - /leonardo_scratch/fast/ICT24_MHPC/ntilinin/INPUT/access-om2/ocean/tides/global.1deg/2020.05.30
          scheduler: slurm
          project: ICT24_MHPC
          walltime: 00:30:00
          queue: boost_usr_prod
          ncpus: 256
          nnodes: 8
          npernode: 32

I’ll try to work around, but would be grateful for any advice as usual

Many thanks!!!

anton · 24 November 2024 21:53

Hi Natalia

I would try commenting out these lines:

github.com

payu-org/payu/blob/27aac372dba2752e1d58ab1d66db799e7891e103/payu/experiment.py#L606-L608


      
          if model_npernode % 2 == 0:
              npernode_flag = ('-map-by ppr:{0}:socket'
                               ''.format(model_npernode / 2))

For reasons that are not clear to me, for some reason payu is requesting 16 tasks per available “socket”, when its probably only possible to have one. Maybe this is a gadi specific detail for some specific case. You might be able to remove the -map-by argument entirely. I think it will take some experimentation.

Natalia · 24 November 2024 22:09

Hi Anton,

Will try, thank you.
It could be that each gadi’s node has 3 sockets each with 16 cores, this makes sense - 16*3=48 cores.

Topic		Replies	Views
Perturbation experiments forked from ACCESS-OM2-025 RYF control run on ik11 Ocean help , access-om2 , solved	20	78	5 November 2024
PAYU issues on Setonix Technical payu	17	409	26 September 2023
Running ACCESS-OM2 on Leonardo supercomputer Technical help , access-om2 , inscope	42	521	17 November 2024
Payu: a workflow manager for some ACCESS models ACCESS-NRI Releases nci , release , payu , nri-updates	5	626	27 August 2024
ACCESS-OM2 payu tutorial COSIMA access-om2 , tutorial , payu	0	636	24 January 2024

Related topics