# ACCESS-ESM1.5 running second year with payu deletes the forcing files for WOMBAT

**URL:** <https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114>\
**Category:** Earth System Model\
**Tags:** help, payu\
**Created:** [22 May 2024 01:40 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114 "2024-05-22T01:40:38Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![pearseb](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pearseb/32/1033_2.png) [@pearseb](https://forum.access-hive.org.au/u/pearseb)\
**Post date:** [22 May 2024 01:40 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/1 "2024-05-22T01:40:39Z")

</div>

Hi all,

@matthew.chamberlain and I have managed to get ACCESS-ESM1.5 working with the new WOMBAT.

We run with payu, and the first year runs perfectly. No problem. All biogeochemical outputs look good.

When I try to run the second year, payu for some reason now deletes all the forcing input files that WOMBAT needs: dust, river inputs, etc… Can someone help me to figure out why payu now deletes these files in the “work” directory?

Thanks all,  
pearse

---

<div class="post-metadata">

**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\
**Post date:** [22 May 2024 02:05 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/2 "2024-05-22T02:05:44Z")

</div>

ACCESS-NRI generally ask that you add the #help hashtag to your post.

---

<div class="post-metadata">

**Author:** ![ezhilsabareesh8](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ezhilsabareesh8/32/790_2.png) [@ezhilsabareesh8](https://forum.access-hive.org.au/u/ezhilsabareesh8)\
**Post date:** [22 May 2024 04:24 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/3 "2024-05-22T04:24:13Z")

</div>

Thanks @pearseb for reaching out. To better assist you, could you please provide some additional information.

1. The steps you followed to set up and run the second year.
2. Any error messages or logs generated when you try to run the second year.
3. What version of payu are you using?
4. Where is your control directory located, and are the permissions set correctly to ensure it is accessible?

---

<div class="post-metadata">

**Author:** ![pearseb](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pearseb/32/1033_2.png) [@pearseb](https://forum.access-hive.org.au/u/pearseb)\
**Post date:** [23 May 2024 22:25 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/4 "2024-05-23T22:25:42Z")

</div>

Hi @ezhilsabareesh8,

My **steps** are:

1. payu sweep
2. payu setup
3. payu run -f

My **config.yaml file** looks like this:

```yaml
jobname: pi_access1.6
#project: p66 
queue: normal
walltime: 3:10:00
  
# note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER
laboratory: access-esm1.6
model: access
  
submodels:
    - name: atmosphere
      model: um
      ncpus: 192 
      exe: /g/data/access/payu/access-esm/bin/coe/um7.3x
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere
        - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
 
    - name: ocean
      model: mom 
      ncpus: 180 
      #exe: /g/data/access/payu/access-esm/bin/coe/mom5xx
      #exe: /g/data/p66/mtc599/access/may23/MOM5-matt-csiro/exec/nci/ACCESS-ESM/fms_ACCESS-ESM.x 
      exe: /g/data/p66/mtc599/access/may24/MOM5/exec/nci/ACCESS-ESM/fms_ACCESS-ESM.x
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common
        #- /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
        - /g/data/p93/pjb581/access-esm/files_pre-industrial
 
    - name: ice 
      model: cice
      ncpus: 12
      exe: /g/data/access/payu/access-esm/bin/coe/cicexx
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ice
 
    - name: coupler
      model: oasis
      ncpus: 0
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/coupler
 
collate:
   exe: /g/data/access/payu/access-esm/bin/mppnccombine
   restart: true
   mem: 4GB 
 
#restart: /g/data/access/payu/access-esm/restart/pre-industrial
restart: /g/data/es60/pjb581/access_esm/restart/PI_lite
 
calendar:
    start:
        year: 101 
        month: 1
        days: 1
 
    runtime:
        years: 1
        months: 0
        days: 0
 
runspersub: 1
 
stacksize: unlimited
 
qsub_flags: -W umask=027

```

I have access permissions to all the above directories and files. The first year of the simulation runs absolutely fine with the same config. No problem.

The **error** in the .err file is clear when trying to run the second year:  
FATAL from PE 179: MPP\_OPEN:INPUT/dust.nc does not exist.  
When I look in the work directory, the error is confirmed. No dust.nc file exists there, even though it does after I run “payu setup”.

**payu version** = payu 1.0.19

Thanks for your help,  
pearse

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [24 May 2024 03:31 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/5 "2024-05-24T03:31:47Z")

</div>



---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [24 May 2024 04:05 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/6 "2024-05-24T04:05:17Z")

</div>

Hi @pearseb

Looking at the CLEX pre-industrial configuration (which I assume this is based on) those files are in `/g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial`:

> <https://github.com/coecms/access-esm/blob/pre-industrial/manifests/input.yaml#L359-L363>

In the `config.yaml` you provided above you have commented out that input directory and replaced it with your own:

```yaml
      input:
        - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common
        #- /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
        - /g/data/p93/pjb581/access-esm/files_pre-industrial

```

Do the files you need exist in `/g/data/p93/pjb581/access-esm/files_pre-industrial`? If not you can either copy them into your directory, or you can uncomment that input line. `payu` [will overwrite any existing symlinks](https://github.com/payu-org/payu/blob/master/payu/fsops.py#L141-L144) if the same files exist in your input directory. In this way any files that aren’t found in your input directory will be picked up from `/g/data/access/`.

[Note: that is my recollection, and what the code suggests, but you can ensure this is working correctly by doing `payu setup` and check the correct files are being symlinked into `work/ocean/INPUT`]

Now you say when you do `payu setup` you can see the files you need, but when you run the model it says they are missing? Best to check that they are in `work/ocean/INPUT` when symlinked as [that is where the model expects to find them](https://github.com/mom-ocean/MOM5/blob/d7ba13a3f364ce130b6ad0ba813f01832cada7a2/src/mom5/ocean_csiro_bgc/csiro_bgc.F90#L1582).

It may work the first time if the missing files are in `/g/data/es60/pjb581/access_esm/restart/PI_lite`, as you have [specified that as a `restart` directory](https://payu.readthedocs.io/en/latest/config.html#miscellaneous), so all the contents of that directory will be symlinked into `work/ocean/INPUT` **the first time you run the model** , but not on subsequent runs.

If that isn’t the problem then I think you’ll need to give me access to your control directory (where your `config.yaml` is located).

---

<div class="post-metadata">

**Author:** ![pearseb](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pearseb/32/1033_2.png) [@pearseb](https://forum.access-hive.org.au/u/pearseb)\
**Post date:** [24 May 2024 06:40 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/7 "2024-05-24T06:40:41Z")

</div>

Hi @Aidan,

Thanks for looking into this for me. In answer to your questions:

1. Do the files you need exist in `/g/data/p93/pjb581/access-esm/files_pre-industrial` ? **yes, but they exist as symlinks**
2. Best to check that they are in `work/ocean/INPUT` when symlinked as [that is where the model expects to find them](https://github.com/mom-ocean/MOM5/blob/d7ba13a3f364ce130b6ad0ba813f01832cada7a2/src/mom5/ocean_csiro_bgc/csiro_bgc.F90#L1582). **they are after I run “payu setup”**
3. It may work the first time if the missing files are in `/g/data/es60/pjb581/access_esm/restart/PI_lite`. **They are not**

To restate the problem:  
I run _payu setup_ → All files are accounted for in `/work` → I run _payu run -f_ → the symlinks from `/g/data/p93/pjb581/access-esm/files_pre-industrial` are deleted and the model crashes.

The only thing I can think of now is that in (1) above I shouldn’t have symlinks in `/g/data/p93/pjb581/access-esm/files_pre-industrial` but real files.

Thanks for your help Aidan!  
pearse

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [24 May 2024 07:08 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/8 "2024-05-24T07:08:14Z")

</div>

> [@pearseb](#):
>
> The only thing I can think of now is that in (1) above I shouldn’t have symlinks in `/g/data/p93/pjb581/access-esm/files_pre-industrial` but real files.

If things look ok when you do `payu setup` from a login node, and then stuff up when the jobs is submitted then the first thing I’d look for is if a project code wasn’t being included in the `-l storage` flags when the model is run. `payu` tries to figure out all the projects that are being used and add them to the `storage` flags for you. And running `payu setup` is a great way to help, as it writes the location of all the input, executable and restart files to the manifest files, and `payu` uses that as one of the sources of information.

The simplest thing to do is just copy those files to your input directory rather than symlink them, or remove the symlinks from your input directory and add back the `access` directory and let them be found there, and see if that helps.

You can tell `payu` project codes to add the `storage` flags, but it shouldn’t be necessary in this case.

Give that a crack and see if it helps. If not you’ll either have to give me access to your control directory, or set up a time when we can VC and I can check it out in real time.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [24 May 2024 07:27 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/9 "2024-05-24T07:27:59Z")

</div>

You can see what storage flags are being used when you do `payu run` the full PBS submit command is echo’ed to the screen.

If you want try running again and copy and paste what is printed to the screen and I can help to translate it for you.

---

<div class="post-metadata">

**Author:** ![pearseb](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pearseb/32/1033_2.png) [@pearseb](https://forum.access-hive.org.au/u/pearseb)\
**Post date:** [27 May 2024 01:11 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/10 "2024-05-27T01:11:22Z")

</div>

Hi @Aidan,

Thanks for helping. Instead of using symbolic links, I just copied the actual files into the directory I was pointing to in the config.yaml file. Now things seem to be working. Odd though that it appears to delete things when the directory itself has symbolic links. But, it’s working, and that’s the important thing.  
Thanks for your help!  
pearse

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [28 May 2024 10:13 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/11 "2024-05-28T10:13:52Z")

</div>



---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [29 July 2024 12:24 UTC](https://forum.access-hive.org.au/t/access-esm1-5-running-second-year-with-payu-deletes-the-forcing-files-for-wombat/2114/13 "2024-07-29T12:24:00Z")

</div>