Custom restarts and inputs to release-1deg_jra55_iaf_bgc experiment

COSIMA
, , , ,
1

Hi all,

I “payu cloned” the release-1deg_jra55_iaf_bgc experiment into my home environment on Gadi.

I am wanting to run a hindcast experiment (1958-2020) with the new WOMBAT-lite for CO2 flux assessment. It is the final part of the model assessment before I submit the work to a journal for publication.

I have altered the config.yml file to reflect that WOMBAT-lite comes with:

  • A new executable for the ocean model (I also took the liberty of replacing the executables for the atmosphere, ice and mppcombine with my own)
  • New input files that are read by the model (hydrothermal Fe, river fluxes)
  • New parameters and parameter values (bgc_param.nc)

These changes look like this within config.yaml:

image
image1920×1046 335 KB

I then run “payu sweep”, “payu setup” and “payu run -f”.

Payu sweep followed by payu setup gives the following output:

image
image1920×963 199 KB

Then, when I try to run the model with payu run -f, I get this error in the error file:

payu: error: Input directory /g/data/es60/pjb581/liteexps/lite2optim21/restart020/ocean/csiro_bgc.res.nc not found; aborting.

I am absolutely certain that the path to this file is correct. Why can’t payu access this file?

Thanks,
pearse

2

When you do payu run, does the PBS command include es60 in the storage directory ?

e.g. when I payu run :

laboratory path:  /scratch/tm70/as2285/access-om3
binary path:  /scratch/tm70/as2285/access-om3/bin
input path:  /scratch/tm70/as2285/access-om3/input
work path:  /scratch/tm70/as2285/access-om3/work
archive path:  /scratch/tm70/as2285/access-om3/archive
Metadata and UUID generation is disabled. Experiment name used for archival: MOM6-CICE6
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=00:05:00 -l ncpus=240 -l mem=960GB -l jobfs=10GB -N 1deg_jra55do_ry -l wd -j n -v PAYU_PATH=/g/data/vk83/prerelease/apps/payu/dev/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/tm70/as2285/spack0.22/release/modules/linux-rocky8-x86_64:/etc/scl/modulefiles:/g/data/vk83/prerelease/modules:/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/tm70+gdata/vk83 -- /g/data/vk83/prerelease/apps/payu/dev/bin/python /g/data/vk83/prerelease/apps/payu/dev/bin/payu-run
127007822.gadi-pbs

it includes : -l storage=gdata/tm70+gdata/vk83 in the qsub line

3

Ah. No it doesn’t!

4

Payu should auto-detect the storage and set them, I am not sure why/when it doesn’t. (Possibly its confused by getting an input from a restart directory?)

You could try adding

storage:
     gdata:
              - es60

To your config.yaml

(ref: Configuring your experiment — payu 1.1.5+0.g78f67ff.dirty documentation)

Its probably also worth checking the file is listed in manifests/input.yaml

1 Like
5

Hi Anton,

Thanks for the help.

Now that the model is finding the /g/data/es60 files, it is now saying the manifest files are wrong. Do I manually alter these? And if so, how do I alter the hashes and md5?

image
image2262×724 247 KB

6

As you changed the executable, that is expected.

If you change manifest:reproduce:exe: to False payu will recalculate

Once the executable version has stabilized, you should set this back to True for any sort of run with answers you want to keep

1 Like
7

It’s running! Awesome. Thank you so much Anton.

1 Like
8

But @anton, by now turning the manifest options to True, the model now won’t run because it is saying:

image
image2266×586 215 KB

I was able to run 1 year successfully, then I turned the manifest options back to True and I got the above error.

The model is now running again because I set these options back to False and I think is starting from the right restart file and has the appropriate yearly forcings…

image
image1884×158 39.7 KB

9

Thats odd … we are reaching the limits of my payu knowledge here. I don’t understand why the input.yaml is looking in a restart directory.

So you had an archive directory already, and then just did payu sweep && payu run ?

10

Are you using manifest:reproduce:input:True?
Maybe there’s no way of enabling that (ie changing config.yaml) without failing the input repro test? Catch-22?

1 Like
11

It’s fine to turn off the reproduce options if it works for you. payu is still tracking all the inputs, restarts and executables, just not checking if they change from the previous run.

It may be there is some odd behaviour here we’ve not properly accounted for, but @jo-basevi also recently updated the way the manifest reproduce flag worked, so this behaviour might be fixed in a recent version (1.1.5), or perhaps it was introduced with that change. Hard to say without knowing what payu version is being used.

1 Like