Payu setup hanging for a looooong time

Technical
,
1

Hi all,

just today, i have noticed that payu setup is hanging for a long time at this particular point:

Checking exe, input and restart manifests

it has happened for me in two different configurations using ACCESS-OM3 and specifically in configurations using the ERA5 forcing. This includes the regional-mom6 in its “out-of-the-box” configuration of the Tasmanian domain. It has also happened with Robin van Dijk, who is running OM3 in the single-column configuration.

It does not, however, occur when using OM3 with JRA55 forcings.

Please help on this Friday afternoon (or enjoy your weekend and see you monday).
pearse

3 
[pjb581@gadi-login-07 pjb581]$ payu --version
payu 1.3.6

and my config.yaml file:

# Payu configuration

# If submitting to a different project to your default, uncomment line below 
# and change project code as appropriate; also set shortpath below
# project: x77
# Force payu to always find, and save, files in this scratch project directory
# shortpath: /scratch/v45

# Using payu sync is recommend to copy data from ephemeral scratch space to
# longer term storage
sync:
    enable: false   # set path below and change to true
    base_path:      # Set to location on /g/data (e.g. /g/data/PROJECT/USER/)
    restarts: true
    exclude_uncollated: false

# Model software version
modules:
    use:
        - /g/data/vk83/modules
    load:
        - access-om3/2026.05.004

queue: normalsr
ncpus: 208
jobfs: 10GB
walltime: 04:00:00
jobname: access-rom3-demo-bgc
mem: 512GB

model: access-om3
exe: access-om3-MOM6
input:
    - /g/data/vn19/pjb581/rom3-inputdirs/access-rom3-demo-bgc
    # drof/datm forcing data
    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2026.01.21/JRA55do-drof-ESMFmesh.nc
    - /g/data/qv56/replicas/input4MIPs/CMIP6Plus/OMIP/MRI/MRI-JRA55-do-1-6-0/land/
    - /g/data/qv56/replicas/input4MIPs/CMIP6Plus/OMIP/MRI/MRI-JRA55-do-1-6-0/landIce/
    - /g/data/vk83/prerelease/configurations/inputs/access-om3/share/meshes/share/2026.07.30/ERA5-datm-ESMFmesh.nc
    - /g/data/av17/access-nri/OM3/era5_rechunked_1h_yearly/

runlog: false
metadata:
    enable: false
manifest:
    reproduce:
        exe: true
        input: false

mpi:
    flags:
        - --mca mpi_yield_when_idle true

collate:
    flags: -n4 -m -r
    exe: mppnccombine
    restart: true
    mpi: false
    walltime: 0:30:00
    mem: 30GB
    ncpus: 1
    queue: normalsr

userscripts:
    archive: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/archive.sh

qsub_flags: -W umask=027

restart_freq: 1YS
payu_minimum_version: 1.3.6
4

Hi Pearse, I encountered a similar delay with the yearly ERA5 forcing. In my case, the input manifest still contained JRA55 entries, so payu started computing full MD5 hashes for about 7.6 TB of ERA5 files. It appeared stuck at “Checking exe, input and restart manifests”, but was actually hashing.

I spent around an hour across interrupted attempts, so I don’t have a completed timing for the initial hashing. After copying the ERA5 entries and their existing hashes from a matching manifest, payu setup completed in 8 minutes 17 seconds. Subsequent setups generally took seconds to a couple of minutes.

Could you check whether your manifests/input.yaml contains the ERA5 entries? The MCW ERA5 input manifest has existing hashes you can reuse if you’re using exactly the same unchanged forcing files. Copy the relevant ERA5 entries while retaining your configuration’s other required inputs.

Otherwise, the initial hashing needs to finish; I’d run that setup inside a PBS job rather than on a login node. Make sure setup completes before submitting the model run—in my case, the incomplete manifest caused gdata/av17 to be omitted from the PBS storage request, resulting in “Input … not found” failures.