ACCESS-CM2 pacemaker set ups

sebmckenna · 22 May 2023 01:51

Hi all,

I am wanting to setup a pacemaker experiment in ACCESS-CM2, using a different restoring climatology for each model year.
I already know how to set up an experiment that restores to the same climatology for each model year over a region mask. This is done by adding the following to
/roses/u-exp/app/coupled/rose-app.conf

[file:$OCN_RUNDIR/INPUT/temp_sfc_restore.nc]
source=dir/to/restorevals.nc

[file:$OCN_RUNDIR/INPUT/restore_mask.nc]
source=dir/to/restoremask.nc

and changing lines in this file
/roses/u-exp/app/mom/rose-app.conf

read_restore_mask=.true.

temp_restore_tscale=x

However I want to change the file that is used to restore at each model year/cycle or use one restoring file that has 50 years of data instead of 12 months to restore

I am hoping someone will be able to assist in setting this up?

It would be similar to the pacemaker setup in this study but using SST restoring https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021GL096887

Thanks in advance!
Sebastian

ars599 · 25 May 2023 08:06

Similar work has been done on ESM, however I forgot how to and need do dig it out. Meanwhile, to make it simple we can just copy Dave’s run (u-bx973):

u-bx973/app/coupled/rose-app.conf

[file:$ICE_RUNDIR/INPUT/HadSST_daily.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Daily/HadSST_360x300_Land0_daily_$THISYEAR.nc

[file:$ICE_RUNDIR/INPUT/HadSST_mask.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Mask_TA1.nc

However, need to be noted that Dave has modify the code:
so that you need to use his suit

u-bx973/suite.rc

cp -rf /home/599/dhb599/ACCESS-CM2/submodels/mom5_access2_TASST $CYLC_SUITE_SHARE_DIR/mom
rsync -r $CYLC_SUITE_SHARE_DIR/mom {{COMPUTE_HOST}}:$CYLC_SUITE_SHARE_DIR

If you can not access P66, I have copied to

/scratch/public/ars599/u-bx944d

Let me know if you need more info.

sebmckenna · 26 May 2023 01:20

Thanks so much for this!
I’ll take a look and let you know how it goes in the next few days.

I am assuming that this means the restoring SST file needs to be labelled in this format where $THISYEAR is the model year? So if I change this the directory and file name to where I have my specific yearly climatology files as long as it has $THISYEAR it should work right?

My restoring files would be monthly, so I assume they would work, or do they need to be daily?

ars599 · 27 May 2023 00:47

monthly should work.

sebmckenna · 30 May 2023 01:23

Hi,
Thanks again
The bx973 suite looks like it is a branched historical experiment. My other pacemaker experiment is branched from a piControl experiment.

I am wondering if there is much I need to change, to ensure that runs are identical except for the SST restoring values? just turn off time varying gas mmr and change restart files/dates?

In other suites I have used, there is usually a warm restart directory, start date etc under Run Initialisation and cycling, but in this suite that I have copied it seems to be under jinja2 but this also has other options like things for ozone which I am not sure is the same as the other suite.

This is side by side view of the suites left is bx973 copy I am modifying and right is my first experiment that has run (cw323)

I am also wondering, what is the restoring timescale in this suite, or do you know how to check/change?

Let me know what if you can answer any of this, thanks!

Cheers,
Sebastian

sebmckenna · 30 May 2023 01:48

Also, for the SST restoring file, what should the time dimension look like? I can’t check Daves files as I don’t have permission. This is what I have at the moment:

variables:
        double TIME(TIME) ;
                TIME:long_name = "time" ;
                TIME:units = "days since 0000-01-01 00:00:00" ;
                TIME:time_origin = "01-JAN-0000 00:00:00" ;
                TIME:modulo = 365.242492675781 ;
                TIME:axis = "T" ;
                TIME:calendar = "gregorian" ;

ars599 · 31 May 2023 23:45

if based on piControl, your changes are shown as following:

suite.rc

cd $CYLC_SUITE_SHARE_DIR
if [ -d mom ] ; then
  rm -rf mom
fi
module load git
#dhb599: git clone -b CM2-0.1 https://github.com/mom-ocean/MOM5.git $CYLC_SUITE_SHARE_DIR/mom
#dhb599: use local code modified for the SST pacemaker exp
cp -rf /home/599/dhb599/ACCESS-CM2/submodels/mom5_access2_TASST $CYLC_SUITE_SHARE_DIR/mom
rsync -r $CYLC_SUITE_SHARE_DIR/mom {{COMPUTE_HOST}}:$CYLC_SUITE_SHARE_DIR

Dave modified the mom code directly.

app/coupled/rose-app.conf

[file:$ICE_RUNDIR/INPUT/HadSST_daily.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Daily/HadSST_360x300_Land0_daily_$THISYEAR.nc

[file:$ICE_RUNDIR/INPUT/HadSST_mask.nc]
source=/g/data/p66/dhb599/obs_data/HadSST/Mask_TA1.nc

The rest keep using the one from piControl. (follow yours no need to use Dave’t)

as for the starting year ( I guess, you can give a test. Getting old can’t remember everything sorry, but I hope this is the one)

rose-suite.conf

BASIS='18500101'

Aidan · 1 June 2023 08:09

The modified code is in Dave’s home directory which isn’t necessarily accessible to everyone. It should be pushed to GitHub. Probably as a branch on a fork, but even better if it could be incorporated into MOM5 in a way that it could be “turned on” with a namelist option or compiler pre-processor flag.

NB: Arnold, I modified the suite changes to be in code blocks instead of block quotes. That way whitespace is preserved correctly, long lines are rendered better and it is simple to copy the code changes with the in-built copy to clipboard functionality of code blocks. Hope that is ok.

ars599 · 5 June 2023 07:40

Many thanks, Aidan!!! Somehow code blocks are not working with my school email address but my personal one, many thanks for your help.

Dave’s code … mmmm I will have a separate email with you. But many years ago Arrian has done one without changing the code. I need to double-check with her. I forget how to do it.

ars599 · 8 June 2023 06:33

Dear all,

Many thanks to Zoe and Ariaan for their assistance, and a special thanks to Holger @HoWol76HoWol76 for generously creating the instructions on how to run the pacemaker experiment in CM2 without modifying the MOM code (following Dave’s approach).

The idea is to prepare all of your SST files (not just a single file) in advance. Then, when the model runs each year, you submit the corresponding SST file for that specific year along with it.

More details please refer to
Holger’s note:

gist.github.com

https://gist.github.com/HoWol76/bd69a3d6cd3d900665223bfa45d723b1

New_Task_to_Rose-Cylc.md

# Adding a new task to a suite
by Holger Wolff, CLEX CMS

## Issue
In this example, we want to add an additional task before every instance of the coupler to update the sea surface temperature ancillary file.

## Step 0: Make a copy of the ACCESS-CM2 suite
Run this command on accessdev to make a copy of the **ACCESS-CM2** suite:
```bash
rosie copy u-br565

This file has been truncated. show original

@sebmckenna

sebmckenna · 13 July 2023 00:20

Hi thanks for this @holger @ars599
I have a question based on an error I got.

I followed the instructions above in my own suite (u-cy286). In step 3 in the config file you need to define the location of the SST files (FILE_TEMPLATE) and the place that the files are copied to for the model to read from - (SST_FILE).
FILE_TEMPLATE works fine, but the error relates to SST_FILE.

I defined SST_FILE as below.

SST_FILE=$OCN_RUNDIR/INPUT/temp_sfc_restore.nc

This is based off a different suite where there is restoring but SST file doesn’t change.

This ended up giving an error after 1 model cycle, when update_sst runs again. Basically, the directory I am telling the code to copy the new SST file doesn’t exist yet. Here is the job.err

cp: cannot create regular file '/home/561/sm2435/cylc-run/u-cy286/work/09520101/coupled/OCN_RUNDIR/INPUT/temp_sfc_restore.nc': No such file or directory
[FAIL] update_sst.sh <<'__STDIN__'
[FAIL]
[FAIL] '__STDIN__' # return-code=4
2023-07-12T23:17:08Z CRITICAL - failed/EXIT

So, the script failed when it tried to copy the file to the location of SST_FILE
The job.out file suggests that the update_sst script did not fail when it tested this directory in the start of the script which is weird, I think it must only test if you have set the variable, maybe better to make it test whether the directory exists?

So I would like to know

Does the location of SST_FILE matter? can it just be the same directory for each (I don’t think so, as looking at the previous run cycle of other run and these input files all appear in a similar spot)
How can we fix this to make sure that these directories are made prior to this running?

I am thinking maybe we can add a line in the update_sst.sh script that creates the directory if it doesn’t exist, but not sure if this will cause issues e.g.

mkdir -p ${SST_FILE}

Let me know your thoughts
Thanks!
Sebastian

sebmckenna · 13 July 2023 02:21

Actually, here are the changes I have made to the steps shown above. This appears to be working so far.

update_sst.sh
I added some lines before the copy of the restoring file is made

test -z ${SST_FILE_DIR} && exit 5
mkdir -p ${SST_FILE_DIR}

For this to work, I changed app/update_sst/rose-app.conf, to look like this:

[command]
default=update_sst.sh

[env]
FILE_TEMPLATE=/location/of/restoring/file_YEAR.nc
SST_FILE=$OCN_RUNDIR/INPUT/temp_sfc_restore.nc
SST_FILE_DIR=$OCN_RUNDIR/INPUT
~

And in metadata file app/update_sst/meta/rose-meta.conf I added these lines:

[env=SST_FILE_DIR]
description=directory to send SST File to
help=directory for SST file expected by model
compulsory=true

Making these changes worked so that the next cycle could start without the work/09520101/coupled/OCN_RUNDIR/INPUT/ having been created by the coupled job. The coupled job is now running fine

holger · 13 July 2023 06:20

Hi Sebastian

The process I described in my little how-to assumes that the model expects the Sea Surface Temperatures to be located in a specific file.

The new task then overwrites this file every year with a new one.

So SST_FILE is the file to be overwritten and FILE_TEMPLATE is the template of where to find the file which should become the new sst file (with the text YEAR to be replaced by the actual year).

So say

SST_FILE=/scratch/w40/hxw599/access/expt/um/data/sst.nc
FILE_TEMPLATE=/g/data/w40/hxw599/sst_updates/sst_YEAR.nc

Then when the task runs, it would look for the year, say 1998, and then execute this command:

cp /g/data/w40/hxw599/sst_updates/sst_1998.nc /scratch/w40/hxw599/access/expt/um/data/sst.nc

And then the next year, it would run the command:

cp /g/data/w40/hxw599/sst_updates/sst_1999.nc /scratch/w40/hxw599/access/expt/um/data/sst.nc

thus overwriting the previous file. (Of course the data file on /g/data remains untouched.)

As for your error, I would suspect that the directory /home/561/sm2435/cylc-run/u-cy286/work/09520101/coupled/OCN_RUNDIR/INPUT/ does not exist. I don’t know how best to solve this.

Cheers
Holger