# Tips, tricks, and troubleshooting for new ACCESS-ESM 1.5 user

**URL:** <https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166>\
**Category:** Earth System\
**Tags:** help, payu, access-esm\
**Created:** [17 June 2024 02:20 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166 "2024-06-17T02:20:19Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [17 June 2024 02:20 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/1 "2024-06-17T02:20:20Z")

</div>

Hi all, new ACCESS-ESM user here, so sorry for any silly questions.

I want to run a series of experiments branching from the pre-industrial control where I perturb/warm the Southern Ocean CDW layer. At the start, I’m going to try doing this by just modifying the ocean restart file.

I understand ACCESS NRI will be releasing a supported version soon, but to get started in the meantime, I’ve been following the Quickstart guidelines [here](https://github.com/coecms/access-esm). In doing so, I’ve run into a few problems, which thanks to @HIMADRI_SAINI and @dkhutch are now solved (🤞).

1. You need to be a member of the `access` software project on Gadi to access the atmosphere files. Maybe it’s worth mentioning this in the quickstart guide(?), though perhaps this will be redundant with the ACCESS NRI supported version.

2. I had trouble with the atmosphere restart file as mentioned in this [forum post](https://forum.access-hive.org.au/t/out-of-the-box-access-esm1-5-with-payu-not-working/2075) and I think [relevant here too](https://forum.access-hive.org.au/t/run-access-esm-fails-with-error-code-139/1749/18). This was solved by using the backup restart file in the same directory, but would it be better practice to use the restart here: `/g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/`?

I have a few other queries too:

**Changing the restart file**  
At the moment, the model is running and I’m happy to play around with tests using the existing restart (yr 101). However, I would like to change this when I run my actual experiments by starting from a later year, e.g. yr 700. Does anyone know where these restarts for the pre-industrial control are stored?

If I change the restart year, do I need to be aware of any other intricacies? @dkhutch mentioned I also need to change the datestamp for the atmosphere. Is there anything else I should be modifying?

**Git branches vs new directories**  
I’m a bit confused as to best-practice workflow for running new experiments (at least until the ACCESS NRI version is released). Should I be cloning the Github repo to a new folder for each new experiment, or, if using payu from vk83, should I be using new git branches for each experiment? If I stuff something up with the latter method, could I accidentally overwrite data?

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [17 June 2024 04:21 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/2 "2024-06-17T04:21:29Z")

</div>

> [@hrsdawson](#):
>
> Should I be cloning the Github repo to a new folder for each new experiment, or, if using payu from vk83, should I be using new git branches for each experiment? If I stuff something up with the latter method, could I accidentally overwrite data?

The intention of the new branching features, and automated experiment versioning, is that it should be _harder_ to accidentally overwrite data.

With the way `payu` currently works if you accidentally make an experiment control directory with the same name as one that already exists it will find that laboratory in `/scratch` and use it. Typically this doesn’t result in data loss, but can add to an existing run rather than start a new experiment. This can be a pain to unravel. It can lead to data loss if `payu sweep --clean` is used, which will delete the whole laboratory directory containing outputs, restarts and logs.

To ensure the new branching approach works best make sure to use `payu clone` and `payu branch` rather than `git branch`. `payu` wraps the native `git` commands, but also checks the state of the control directory, and creates a new experiment ID and updates the links to `archive` when it decides this is required.

If you’re unsure exactly how the new features work play around a bit, create a test experiment and use `payu checkout -b <expname>` to create a few experiments, see what `payu` does when a new experiment is created, and how to switch between them.

---

<div class="post-metadata">

**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\
**Post date:** [17 June 2024 06:32 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/3 "2024-06-17T06:32:11Z")

</div>

> [@hrsdawson](#):
>
> However, I would like to change this when I run my actual experiments by starting from a later year, e.g. yr 700. Does anyone know where these restarts for the pre-industrial control are stored?

I would ask @tiloz about this, so tagging him here.

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [17 June 2024 23:37 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/4 "2024-06-17T23:37:42Z")

</div>

Thanks @Aidan, I’ll have a play around. Just to clarify, to use the updated payu version, should I be running `module use /g/data/vk83/modules` and `module load payu/1.1.3` (rather than `module load conda/analysis3` which I currently do) every time I log onto gadi to run more years of an experiment, or start a new one?

---

<div class="post-metadata">

**Author:** ![spencerwong](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/spencerwong/32/1894_2.png) [@spencerwong](https://forum.access-hive.org.au/u/spencerwong)\
**Post date:** [18 June 2024 01:25 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/5 "2024-06-18T01:25:00Z")

</div>



---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [18 June 2024 02:06 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/6 "2024-06-18T02:06:32Z")

</div>

> [@hrsdawson](#):
>
> Just to clarify, to use the updated payu version, should I be running `module use /g/data/vk83/modules` and `module load payu/1.1.3` (rather than `module load conda/analysis3` which I currently do) every time I log onto gadi to run more years of an experiment, or start a new one?

Yes. But there are ways to make this easier. It is pretty safe to add something like this to your `~/.bash_profile`

```bash
if in_interactive_shell && in_login_shell
then
  if $(command -v module > /dev/null 2>&1)
  then
    module use /g/data/vk83/modules 
    module load payu
  fi
fi

```

Which will mean `payu` is automatically available every time you log onto `gadi`.

If you don’t want to always have it loaded you can define an alias and then it is a single command to have `payu` available:

```bash
$ alias loadpayu='module use /g/data/vk83/modules && module load payu'
$ module list
Currently Loaded Modulefiles:
 1) pbs 2) dot 3) ncview/2.1.7 4) netcdf/4.7.3(default) 5) git/2.37.3  
$ loadpayu 
$ module list
Currently Loaded Modulefiles:
 1) pbs 2) dot 3) ncview/2.1.7 4) netcdf/4.7.3(default) 5) git/2.37.3 6) payu/1.1.3  

```

You can add that `alias` command to your `~/.bashrc` and then `loadpayu` will always be available on `gadi`.

---

<div class="post-metadata">

**Author:** ![spencerwong](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/spencerwong/32/1894_2.png) [@spencerwong](https://forum.access-hive.org.au/u/spencerwong)\
**Post date:** [23 June 2024 23:40 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/8 "2024-06-23T23:40:57Z")

</div>

Hi Hannah, great questions!

> [@hrsdawson](#):
>
> This was solved by using the backup restart file in the same directory, but would it be better practice to use the restart here: `/g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/`?

Some of the restart files in `/g/data/access/payu/access-esm/restart/pre-industrial` have unclear origins and they [don’t appear to match](https://github.com/coecms/access-esm/issues/11#issue-2135487421) the restart files from the original pre-industrial simulations. For reproducibility, it might be best to use the restarts in `/g/data/vk83`, which are copied from the original `PI-02` simulation.

> [@hrsdawson](#):
>
> Does anyone know where these restarts for the pre-industrial control are stored?

They’re a bit hard to find at the moment, but restart and history files for the CMIP6 experiments along with several non-CMIP experiments are stored in `/g/data/p73/archive`. There are two different ESM1.5 pre-industrial experiments stored there: `PI-01` which was originally run on raijin, and `PI-02` which was run on gadi. `PI-01` isn’t reproducible on gadi, and so it’s probably worth using the restarts from `PI-02` stored in `/g/data/p73/archive/CMIP6/ACCESS-ESM1-5/PI-02`.

> [@hrsdawson](#):
>
> If I change the restart year, do I need to be aware of any other intricacies?

There are a few intricacies! I’ll run through what worked for me in changing the restart year to 700. A few points aren’t completely clear to me still – I’ll update them if I find out anything more. If you or anyone notices anything that’s not quite right, let me know!

1. Copy the relevant restart files from `/g/data/p73/archive/CMIP6/ACCESS-ESM1-5/PI-02/restart` (in this case, those ending with `-07000101` or `-06993112`) to a new directory, e.g. `<new-restart-dir>`, separating them into `atmosphere`, `ocean`, `ice`, and `coupler` subdirectories.

2. In the experiment’s `config.yaml` file, edit the `restart` option to point to `<new-restart-dir>`.

3. Create symlinks: Each model will look for restart files which don’t have date-stamps at the end of their name. E.g. in `<control-directory>/ocean/field_table` , the ocean model will try to use:

4. Next, we need to update each model’s namelist files in `<control-directory>/{atmosphere,ocean,ice}` to have the correct starting dates:

Let me know if you have any questions about those steps, or if anything doesn’t work when trying them out.

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [1 July 2024 06:52 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/9 "2024-07-01T06:52:07Z")

</div>

Thanks @spencerwong for these details and instructions. It’s super helpful, I really appreciate it! I’ve been caught up with other things, but trying this out is my to-do list this week. I’ll let you know how I go.

---

<div class="post-metadata">

**Author:** ![spencerwong](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/spencerwong/32/1894_2.png) [@spencerwong](https://forum.access-hive.org.au/u/spencerwong)\
**Post date:** [10 July 2024 00:08 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/10 "2024-07-10T00:08:07Z")

</div>

No worries! Step 6 turns out to be uneccessary as long as you run `payu setup`, and so I’ve just updated the notes.

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [22 July 2024 00:24 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/11 "2024-07-22T00:24:44Z")

</div>

@spencerwong these instructions are legendary, thank you!!! I finally got around to changing the restart year last week and this worked perfectly for me, no hiccups at all! 🙌

Also, @Aidan I’ve been using the access nri payu version and it’s working a charm. I like the branching feature for keeping track of different simulations, without having multiple run directories.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [25 July 2024 06:01 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/12 "2024-07-25T06:01:53Z")

</div>

> [@hrsdawson](#):
>
> Also, @Aidan I’ve been using the access nri payu version and it’s working a charm. I like the branching feature for keeping track of different simulations, without having multiple run directories.

Awesome! Thanks for the feedback @hrsdawson. It is really good to know.

Your hard work is being used and valued @jo-basevi!

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [5 August 2024 06:58 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/13 "2024-08-05T06:58:40Z")

</div>

@spencerwong what did you use to calculate the total number of seconds between the restart date and the initialisation date using the proleptic Gregorian calendar? I want to branch from other years of the piControl but am unsure I’ve got the correct runtime0 values.

---

<div class="post-metadata">

**Author:** ![spencerwong](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/spencerwong/32/1894_2.png) [@spencerwong](https://forum.access-hive.org.au/u/spencerwong)\
**Post date:** [5 August 2024 07:42 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/14 "2024-08-05T07:42:33Z")

</div>

Hi Hannah, I’m glad that the instructions worked!

I’ve been using the `cftime` library in python which I’ve found useful for calculations with different calendars:

```python
>>> import cftime
>>> init_date = cftime.datetime(1,1,1,0,0,0, calendar = "proleptic_gregorian")
>>> end_date = cftime.datetime(101,1,1,0,0,0, calendar = "proleptic_gregorian")
>>> (end_date - start_date).total_seconds()
3155673600.0

```

I think this library should be available on gadi through either the `vk83/payu` module or otherwise the `hh5` `conda/analysis3` module.

Also as a side note – we’re hoping to simplify cice’s restart date handling for ESM1.5, as the current fairly convoluted form is quite difficult to work with and has previously led to bugs. Further down the line we’re hoping to make the whole process of setting up the model with a different restart a bit simpler. I’ll update the notes when those changes happen!

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [5 August 2024 10:42 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/15 "2024-08-05T10:42:58Z")

</div>

Thank you @spencerwong, thats great!!

---

<div class="post-metadata">

**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\
**Post date:** [27 August 2024 23:26 UTC](https://forum.access-hive.org.au/t/tips-tricks-and-troubleshooting-for-new-access-esm-1-5-user/2166/16 "2024-08-27T23:26:02Z")

</div>