Porting CSIRO/UMUI ACCESS-ESM1.5 ksh run script to payu

Earth System
1

Many of the CMIP6 experiments for ACCESS-ESM1.5 were not run with Rose/CYLC or payu, but rather a series of heavily modified Korn shell (ksh) scripts that were originally generated by the old UMUI configuration program. In the future, payu is looking like the preferred way to run ACCESS-ESM1.5 because the integration with git, the input file manifests and the run history all help make running experiments easy, well documented and reproducible. It’s therefore highly desired that there is some way to port existing configurations of ACCESS that use the ksh scripts to payu. There are some existing payu configurations of CMIP6 experiments (pre-industrial, historical and ssp585), but none of them can exactly reproduce the results of the output from the ksh scripts (as far as I know).

I have attempted to do this for one of my own experiments simply because I’m familiar with them:

payu

ksh

If I’m successful at reproducing my results then I hope the procedure could be easily done for other experiments. So far, the configuration runs, but does not exactly reproduce my results. Reasons for this might be:

  • I missed something/made some mistake
  • There are configuration capabilities in payu that do not exist or are not explicit in the ksh scripts
  • There are configuration capabilities in the ksh scripts that do not exist in payu

Procedure

  1. I ran the ksh scripts my chosen experiment (not submitted to the queue). Most of the interesting configuration is done in this file. It produces a run directory with all of the required namelists for the experiment.
  2. Copy over the atnosphere configuration in tmp_ctrl. Payu lists the namelist and other configuration files it needs here. Some of these have slightly different names to the ksh script. I copied the files over to the relevant files. The file namelists has namelists from all files (so I’m not sure what the point of having all the other namelist files is…) I just made sure they are all consistent.
CNTLALL => CNTLALL
CNTLATM => prefix.CNTLATM
CNTLGEN => prefix.CNTLGEN
CONTCNTL => CONTCNTL
INITHIS => INITHIS
PRESM_A => prefix.PRESM_A
SIZES => SIZES
STASHC => STASHC
UAFILES_A => UAFILES_A
UAFLDS_A => UAFLDS_A
../cable.nml => cable.nml
../input_atm.nml => input_atm.nml
  1. I did the same thing as the above step for the ocean and sea ice files and namcouple
  2. I set the model executable in config.yaml using the ksh executable listed here for the atmosphere, ocean and ice models. Is there a different coupler executable? I couldn’t find this info in the ksh scripts.
  3. Set other config.yaml variables like jobname, calendar: start: year:
  4. This experiment warm-starts from an existing experiment, so I set the details of that in warm-start.sh (year 500 of PI-GWL-t6)
  5. This particular experiment had land-use change enabled. The ksh scripts inject the new land-use map into the restart file at the end of the year, ready for the next year when the job is resubmitted. payu, however, does this before the start of the year in scripts/pre.sh. I had already injected the new land-use map into the restart file, therefore I needed to modify the script to not do this in the first year.
This step was specific to my experiment
  1. Point the warm-start-csrio.sh script to my custom restart file.
  1. Remove references to HISTORY. Some of the config files referenced a paths to HISTORY, which is not used by payu. I searched for these with grep and removed them.
  2. Add the ancil files to atmosphere/um_env.py. The ksh scripts conditionally set these based on the time period (I think…). In my case, most of these were the pre-industrial ancil files. There are some ancil files in um_env.py that I couldn’t find in the ksh scripts. Like ARCLBIOG?
2

To compare results, I converted the output to netcdf with ACCESS-Archiver and did a cdo sub so I can see where the differences are. This takes a while to do. Is there any way to compare the UM binary output files directly?

3

Thanks for documenting this process @tammasloughran.

Just so others know and find the correct repo, all the CLEX CMS ACCESS-ESM1.5 experiments are in branches in this repo:

My recollection was that the payu versions did reproduce the original experiments, but my memory is terrible and I wasn’t directly involved in most of the verification. @holger and/or @Scott may be better placed to say for sure.

They would also be well placed to comment on the UM configuration.

Checking bitwise reproducibility is possible just by comparing the checksums in the UM logs. There is comment here from @MartinDix about this

which also mentions mule-cumf , which maybe answers your other question about comparing fields?

If you want ACCESS-NRI assistance with this add a help tag to the post and it’ll be picked up by the triage team.