# ACCESS-CM2 pacemaker set ups

**URL:** <https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786>\
**Category:** Coupled Model\
**Tags:** access-cm2, pacemaker\
**Created:** [22 May 2023 01:51 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786 "2023-05-22T01:51:19Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [22 May 2023 01:51 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/1 "2023-05-22T01:51:19Z")

</div>

Hi all,

I am wanting to setup a pacemaker experiment in ACCESS-CM2, using a different restoring climatology for each model year.  
I already know how to set up an experiment that restores to the same climatology for each model year over a region mask. This is done by adding the following to  
`/roses/u-exp/app/coupled/rose-app.conf`

```auto
[file:$OCN_RUNDIR/INPUT/temp_sfc_restore.nc]
source=dir/to/restorevals.nc

[file:$OCN_RUNDIR/INPUT/restore_mask.nc]
source=dir/to/restoremask.nc

```

and changing lines in this file  
`/roses/u-exp/app/mom/rose-app.conf`

```auto
read_restore_mask=.true.

temp_restore_tscale=x

```

However I want to change the file that is used to restore at each model year/cycle or use one restoring file that has 50 years of data instead of 12 months to restore

I am hoping someone will be able to assist in setting this up?

It would be similar to the pacemaker setup in this study but using SST restoring [https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021GL096887](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021GL096887)

Thanks in advance!  
Sebastian

---

<div class="post-metadata">

**Author:** ![ars599](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ars599/32/821_2.png) [@ars599](https://forum.access-hive.org.au/u/ars599)\
**Post date:** [25 May 2023 08:06 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/2 "2023-05-25T08:06:29Z")

</div>

Similar work has been done on ESM, however I forgot how to and need do dig it out. Meanwhile, to make it simple we can just copy Dave’s run (u-bx973):

### u-bx973/app/coupled/rose-app.conf

> [file:$ICE\_RUNDIR/INPUT/HadSST\_daily.nc]  
> source=/g/data/p66/dhb599/obs\_data/HadSST/Daily/HadSST\_360x300\_Land0\_daily\_$THISYEAR.nc

> [file:$ICE\_RUNDIR/INPUT/HadSST\_mask.nc]  
> source=/g/data/p66/dhb599/obs\_data/HadSST/Mask\_TA1.nc

However, need to be noted that Dave has modify the code:  
so that you need to use his suit

### u-bx973/suite.rc

> cp -rf /home/599/dhb599/ACCESS-CM2/submodels/mom5\_access2\_TASST $CYLC\_SUITE\_SHARE\_DIR/mom  
> rsync -r $CYLC\_SUITE\_SHARE\_DIR/mom {{COMPUTE\_HOST}}:$CYLC\_SUITE\_SHARE\_DIR

If you can not access P66, I have copied to

> /scratch/public/ars599/u-bx944d

Let me know if you need more info.

---

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [26 May 2023 01:20 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/3 "2023-05-26T01:20:40Z")

</div>

Thanks so much for this!  
I’ll take a look and let you know how it goes in the next few days.

I am assuming that this means the restoring SST file needs to be labelled in this format where $THISYEAR is the model year? So if I change this the directory and file name to where I have my specific yearly climatology files as long as it has $THISYEAR it should work right?

> [@ars599](#):
>
> [file:$ICE\_RUNDIR/INPUT/HadSST\_daily.nc]  
> source=/g/data/p66/dhb599/obs\_data/HadSST/Daily/HadSST\_360x300\_Land0\_daily\_$THISYEAR.nc

My restoring files would be monthly, so I assume they would work, or do they need to be daily?

---

<div class="post-metadata">

**Author:** ![ars599](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ars599/32/821_2.png) [@ars599](https://forum.access-hive.org.au/u/ars599)\
**Post date:** [27 May 2023 00:47 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/4 "2023-05-27T00:47:25Z")

</div>

> [@sebmckenna](#):
>
> k, or do they need

monthly should work.

---

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [30 May 2023 01:23 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/5 "2023-05-30T01:23:42Z")

</div>

Hi,  
Thanks again  
The **bx973** suite looks like it is a branched historical experiment. My other pacemaker experiment is branched from a piControl experiment.

I am wondering if there is much I need to change, to ensure that runs are identical except for the SST restoring values? just turn off time varying gas mmr and change restart files/dates?

In other suites I have used, there is usually a warm restart directory, start date etc under _Run Initialisation and cycling_, but in this suite that I have copied it seems to be under _jinja2_ but this also has other options like things for ozone which I am not sure is the same as the other suite.

This is side by side view of the suites left is bx973 copy I am modifying and right is my first experiment that has run (cw323)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/3a167ac0d8762915b7af0e0bf3d405c45af77f29.png)

I am also wondering, what is the restoring timescale in this suite, or do you know how to check/change?

Let me know what if you can answer any of this, thanks!

Cheers,  
Sebastian

---

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [30 May 2023 01:48 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/6 "2023-05-30T01:48:30Z")

</div>

Also, for the SST restoring file, what should the time dimension look like? I can’t check Daves files as I don’t have permission. This is what I have at the moment:

```auto
variables:
        double TIME(TIME) ;
                TIME:long_name = "time" ;
                TIME:units = "days since 0000-01-01 00:00:00" ;
                TIME:time_origin = "01-JAN-0000 00:00:00" ;
                TIME:modulo = 365.242492675781 ;
                TIME:axis = "T" ;
                TIME:calendar = "gregorian" ;

```

---

<div class="post-metadata">

**Author:** ![ars599](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ars599/32/821_2.png) [@ars599](https://forum.access-hive.org.au/u/ars599)\
**Post date:** [31 May 2023 23:45 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/8 "2023-05-31T23:45:10Z")

</div>

> [@sebmckenna](#):
>
> ring timescale in this suite, or do yo

if based on piControl, your changes are shown as following:

### suite.rc

```auto
cd $CYLC_SUITE_SHARE_DIR
if [-d mom] ; then
  rm -rf mom
fi
module load git
#dhb599: git clone -b CM2-0.1 https://github.com/mom-ocean/MOM5.git $CYLC_SUITE_SHARE_DIR/mom
#dhb599: use local code modified for the SST pacemaker exp
cp -rf /home/599/dhb599/ACCESS-CM2/submodels/mom5_access2_TASST $CYLC_SUITE_SHARE_DIR/mom
rsync -r $CYLC_SUITE_SHARE_DIR/mom {{COMPUTE_HOST}}:$CYLC_SUITE_SHARE_DIR

```

Dave modified the mom code directly.

### app/coupled/rose-app.conf

```auto
[file:$ICE_RUNDIR/INPUT/HadSST_daily.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Daily/HadSST_360x300_Land0_daily_$THISYEAR.nc

[file:$ICE_RUNDIR/INPUT/HadSST_mask.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Mask_TA1.nc

```

The rest keep using the one from piControl. (follow yours no need to use Dave’t)

as for the starting year ( I guess, you can give a test. Getting old can’t remember everything sorry, but I hope this is the one)

### rose-suite.conf

```auto
BASIS='18500101'

```

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [1 June 2023 08:09 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/9 "2023-06-01T08:09:56Z")

</div>

> [@ars599](#):
>
> Dave modified the mom code directly

The modified code is in Dave’s home directory which isn’t necessarily accessible to everyone. It should be pushed to GitHub. Probably as a branch on a fork, but even better if it could be incorporated into `MOM5` in a way that it could be “turned on” with a namelist option or compiler pre-processor flag.

NB: Arnold, I modified the suite changes to be in [code blocks](https://forum.access-hive.org.au/t/forum-user-guide/95) instead of block quotes. That way whitespace is preserved correctly, long lines are rendered better and it is simple to copy the code changes with the in-built copy to clipboard functionality of code blocks. Hope that is ok.

---

<div class="post-metadata">

**Author:** ![ars599](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ars599/32/821_2.png) [@ars599](https://forum.access-hive.org.au/u/ars599)\
**Post date:** [5 June 2023 07:40 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/10 "2023-06-05T07:40:33Z")

</div>

Many thanks, Aidan!!! Somehow code blocks are not working with my school email address but my personal one, many thanks for your help.

Dave’s code … mmmm I will have a separate email with you. But many years ago Arrian has done one without changing the code. I need to double-check with her. I forget how to do it.

---

<div class="post-metadata">

**Author:** ![ars599](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ars599/32/821_2.png) [@ars599](https://forum.access-hive.org.au/u/ars599)\
**Post date:** [8 June 2023 06:33 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/11 "2023-06-08T06:33:27Z")

</div>

Dear all,

Many thanks to Zoe and Ariaan for their assistance, and a special thanks to Holger @HoWol76HoWol76 for generously creating the instructions on how to run the pacemaker experiment in CM2 without modifying the MOM code (following Dave’s approach).

The idea is to prepare all of your SST files (not just a single file) in advance. Then, when the model runs each year, you submit the corresponding SST file for that specific year along with it.

More details please refer to  
Holger’s note:

> <https://gist.github.com/HoWol76/bd69a3d6cd3d900665223bfa45d723b1>

@sebmckenna

---

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [13 July 2023 00:20 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/12 "2023-07-13T00:20:46Z")

</div>

Hi thanks for this @holger @ars599  
I have a question based on an error I got.

I followed the instructions above in my own suite (u-cy286). In step 3 in the config file you need to define the location of the SST files ( **`FILE_TEMPLATE`** ) and the place that the files are copied to for the model to read from - ( **`SST_FILE`** ).  
`FILE_TEMPLATE` works fine, but the error relates to `SST_FILE`.

I defined SST\_FILE as below.

```auto
SST_FILE=$OCN_RUNDIR/INPUT/temp_sfc_restore.nc

```

This is based off a different suite where there is restoring but SST file doesn’t change.

This ended up giving an error after 1 model cycle, when update\_sst runs again. Basically, the directory I am telling the code to copy the new SST file doesn’t exist yet. Here is the job.err

```auto
cp: cannot create regular file '/home/561/sm2435/cylc-run/u-cy286/work/09520101/coupled/OCN_RUNDIR/INPUT/temp_sfc_restore.nc': No such file or directory
[FAIL] update_sst.sh <<' __STDIN__'
[FAIL]
[FAIL] ' __STDIN__' # return-code=4
2023-07-12T23:17:08Z CRITICAL - failed/EXIT

```

So, the script failed when it tried to copy the file to the location of `SST_FILE`  
The job.out file suggests that the update\_sst script did not fail when it tested this directory in the start of the script which is weird, I think it must only test if you have set the variable, maybe better to make it test whether the directory exists?

So I would like to know

1. Does the location of `SST_FILE` matter? can it just be the same directory for each (I don’t think so, as looking at the previous run cycle of other run and these input files all appear in a similar spot)
2. How can we fix this to make sure that these directories are made prior to this running?

I am thinking maybe we can add a line in the update\_sst.sh script that creates the directory if it doesn’t exist, but not sure if this will cause issues e.g.

```auto
mkdir -p ${SST_FILE}

```

Let me know your thoughts  
Thanks!  
Sebastian

---

<div class="post-metadata">

**Author:** ![sebmckenna](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sebmckenna](https://forum.access-hive.org.au/u/sebmckenna)\
**Post date:** [13 July 2023 02:21 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/13 "2023-07-13T02:21:57Z")

</div>

Actually, here are the changes I have made to the steps shown above. This appears to be working so far.

**update\_sst.sh**  
I added some lines before the copy of the restoring file is made

```auto
test -z ${SST_FILE_DIR} && exit 5
mkdir -p ${SST_FILE_DIR}

```

For this to work, I changed `app/update_sst/rose-app.conf`, to look like this:

```auto
[command]
default=update_sst.sh

[env]
FILE_TEMPLATE=/location/of/restoring/file_YEAR.nc
SST_FILE=$OCN_RUNDIR/INPUT/temp_sfc_restore.nc
SST_FILE_DIR=$OCN_RUNDIR/INPUT
~                                  

```

And in metadata file `app/update_sst/meta/rose-meta.conf` I added these lines:

```auto
[env=SST_FILE_DIR]
description=directory to send SST File to
help=directory for SST file expected by model
compulsory=true

```

Making these changes worked so that the next cycle could start without the `work/09520101/coupled/OCN_RUNDIR/INPUT/` having been created by the coupled job. The coupled job is now running fine

---

<div class="post-metadata">

**Author:** ![holger](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/holger/32/320_2.png) [@holger](https://forum.access-hive.org.au/u/holger)\
**Post date:** [13 July 2023 06:20 UTC](https://forum.access-hive.org.au/t/access-cm2-pacemaker-set-ups/786/14 "2023-07-13T06:20:46Z")

</div>

Hi Sebastian

The process I described in my little how-to assumes that the model expects the Sea Surface Temperatures to be located in a specific file.

The new task then _overwrites_ this file every year with a new one.

So `SST_FILE` is the file _to be overwritten_ and `FILE_TEMPLATE` is the template of _where to find the file_ which should become the new sst file (with the text `YEAR` to be replaced by the actual year).

So say

```auto
SST_FILE=/scratch/w40/hxw599/access/expt/um/data/sst.nc
FILE_TEMPLATE=/g/data/w40/hxw599/sst_updates/sst_YEAR.nc

```

Then when the task runs, it would look for the year, say 1998, and then execute this command:

```auto
cp /g/data/w40/hxw599/sst_updates/sst_1998.nc /scratch/w40/hxw599/access/expt/um/data/sst.nc

```

And then the next year, it would run the command:

```auto
cp /g/data/w40/hxw599/sst_updates/sst_1999.nc /scratch/w40/hxw599/access/expt/um/data/sst.nc

```

thus overwriting the previous file. (Of course the data file on `/g/data` remains untouched.)

As for your error, I would suspect that the directory `/home/561/sm2435/cylc-run/u-cy286/work/09520101/coupled/OCN_RUNDIR/INPUT/` does not exist. I don’t know how best to solve this.

Cheers  
Holger