ACCESS-ESM1.5 running second year with payu deletes the forcing files for WOMBAT

Earth System Earth System Model
, ,
1

Hi all,

@matthew.chamberlain and I have managed to get ACCESS-ESM1.5 working with the new WOMBAT.

We run with payu, and the first year runs perfectly. No problem. All biogeochemical outputs look good.

When I try to run the second year, payu for some reason now deletes all the forcing input files that WOMBAT needs: dust, river inputs, etc… Can someone help me to figure out why payu now deletes these files in the “work” directory?

Thanks all,
pearse

2

ACCESS-NRI generally ask that you add the help hashtag to your post.

1 Like
3

Thanks @pearseb for reaching out. To better assist you, could you please provide some additional information.

  1. The steps you followed to set up and run the second year.
  2. Any error messages or logs generated when you try to run the second year.
  3. What version of payu are you using?
  4. Where is your control directory located, and are the permissions set correctly to ensure it is accessible?
4

Hi @ezhilsabareesh8,

My steps are:

  1. payu sweep
  2. payu setup
  3. payu run -f

My config.yaml file looks like this:

jobname: pi_access1.6
#project: p66 
queue: normal
walltime: 3:10:00
  
# note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER
laboratory: access-esm1.6
model: access
  
submodels:
    - name: atmosphere
      model: um
      ncpus: 192 
      exe: /g/data/access/payu/access-esm/bin/coe/um7.3x
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere
        - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
 
    - name: ocean
      model: mom 
      ncpus: 180 
      #exe: /g/data/access/payu/access-esm/bin/coe/mom5xx
      #exe: /g/data/p66/mtc599/access/may23/MOM5-matt-csiro/exec/nci/ACCESS-ESM/fms_ACCESS-ESM.x 
      exe: /g/data/p66/mtc599/access/may24/MOM5/exec/nci/ACCESS-ESM/fms_ACCESS-ESM.x
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common
        #- /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
        - /g/data/p93/pjb581/access-esm/files_pre-industrial
 
    - name: ice 
      model: cice
      ncpus: 12
      exe: /g/data/access/payu/access-esm/bin/coe/cicexx
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ice
 
    - name: coupler
      model: oasis
      ncpus: 0
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/coupler
 
collate:
   exe: /g/data/access/payu/access-esm/bin/mppnccombine
   restart: true
   mem: 4GB 
 
#restart: /g/data/access/payu/access-esm/restart/pre-industrial
restart: /g/data/es60/pjb581/access_esm/restart/PI_lite
 
calendar:
    start:
        year: 101 
        month: 1
        days: 1
 
    runtime:
        years: 1
        months: 0
        days: 0
 
runspersub: 1
 
stacksize: unlimited
 
qsub_flags: -W umask=027

I have access permissions to all the above directories and files. The first year of the simulation runs absolutely fine with the same config. No problem.

The error in the .err file is clear when trying to run the second year:
FATAL from PE 179: MPP_OPEN:INPUT/dust.nc does not exist.
When I look in the work directory, the error is confirmed. No dust.nc file exists there, even though it does after I run “payu setup”.

payu version = payu 1.0.19

Thanks for your help,
pearse

6

Hi @pearseb

Looking at the CLEX pre-industrial configuration (which I assume this is based on) those files are in /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial:

In the config.yaml you provided above you have commented out that input directory and replaced it with your own:

      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common
        #- /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
        - /g/data/p93/pjb581/access-esm/files_pre-industrial

Do the files you need exist in /g/data/p93/pjb581/access-esm/files_pre-industrial? If not you can either copy them into your directory, or you can uncomment that input line. payu will overwrite any existing symlinks if the same files exist in your input directory. In this way any files that aren’t found in your input directory will be picked up from /g/data/access/.

[Note: that is my recollection, and what the code suggests, but you can ensure this is working correctly by doing payu setup and check the correct files are being symlinked into work/ocean/INPUT]

Now you say when you do payu setup you can see the files you need, but when you run the model it says they are missing? Best to check that they are in work/ocean/INPUT when symlinked as that is where the model expects to find them.

It may work the first time if the missing files are in /g/data/es60/pjb581/access_esm/restart/PI_lite, as you have specified that as a restart directory, so all the contents of that directory will be symlinked into work/ocean/INPUT the first time you run the model, but not on subsequent runs.

If that isn’t the problem then I think you’ll need to give me access to your control directory (where your config.yaml is located).

7

Hi @Aidan,

Thanks for looking into this for me. In answer to your questions:

  1. Do the files you need exist in /g/data/p93/pjb581/access-esm/files_pre-industrial ? yes, but they exist as symlinks
  2. Best to check that they are in work/ocean/INPUT when symlinked as that is where the model expects to find them. they are after I run “payu setup”
  3. It may work the first time if the missing files are in /g/data/es60/pjb581/access_esm/restart/PI_lite. They are not

To restate the problem:
I run payu setup → All files are accounted for in /work → I run payu run -f → the symlinks from /g/data/p93/pjb581/access-esm/files_pre-industrial are deleted and the model crashes.

The only thing I can think of now is that in (1) above I shouldn’t have symlinks in /g/data/p93/pjb581/access-esm/files_pre-industrial but real files.

Thanks for your help Aidan!
pearse

8

If things look ok when you do payu setup from a login node, and then stuff up when the jobs is submitted then the first thing I’d look for is if a project code wasn’t being included in the -l storage flags when the model is run. payu tries to figure out all the projects that are being used and add them to the storage flags for you. And running payu setup is a great way to help, as it writes the location of all the input, executable and restart files to the manifest files, and payu uses that as one of the sources of information.

The simplest thing to do is just copy those files to your input directory rather than symlink them, or remove the symlinks from your input directory and add back the access directory and let them be found there, and see if that helps.

You can tell payu project codes to add the storage flags, but it shouldn’t be necessary in this case.

Give that a crack and see if it helps. If not you’ll either have to give me access to your control directory, or set up a time when we can VC and I can check it out in real time.

1 Like
9

You can see what storage flags are being used when you do payu run the full PBS submit command is echo’ed to the screen.

If you want try running again and copy and paste what is printed to the screen and I can help to translate it for you.

10

Hi @Aidan,

Thanks for helping. Instead of using symbolic links, I just copied the actual files into the directory I was pointing to in the config.yaml file. Now things seem to be working. Odd though that it appears to delete things when the directory itself has symbolic links. But, it’s working, and that’s the important thing.
Thanks for your help!
pearse

2 Likes