Many of the CMIP6 experiments for ACCESS-ESM1.5 were not run with Rose/CYLC or payu, but rather a series of heavily modified Korn shell (ksh) scripts that were originally generated by the old UMUI configuration program. In the future, payu is looking like the preferred way to run ACCESS-ESM1.5 because the integration with git, the input file manifests and the run history all help make running experiments easy, well documented and reproducible. It’s therefore highly desired that there is some way to port existing configurations of ACCESS that use the ksh scripts to payu. There are some existing payu configurations of CMIP6 experiments (pre-industrial, historical and ssp585), but none of them can exactly reproduce the results of the output from the ksh scripts (as far as I know).
I have attempted to do this for one of my own experiments simply because I’m familiar with them:
payu
ksh
If I’m successful at reproducing my results then I hope the procedure could be easily done for other experiments. So far, the configuration runs, but does not exactly reproduce my results. Reasons for this might be:
I missed something/made some mistake
There are configuration capabilities in payu that do not exist or are not explicit in the ksh scripts
There are configuration capabilities in the ksh scripts that do not exist in payu
Procedure
I ran the ksh scripts my chosen experiment (not submitted to the queue). Most of the interesting configuration is done in this file. It produces a run directory with all of the required namelists for the experiment.
Copy over the atnosphere configuration in Running.dir/ATM_RUNDIR/tmp_ctrl to the payu config. Payu lists the namelist and other configuration files it needs here. Some of these have slightly different names to the ksh script. I copied the files over to the relevant files. The file namelists has namelists from all files (so I’m not sure what the point of having all the other namelist files is…) I just made sure they are all consistent.
I did the same thing as the above step for the ocean and sea ice files and namcouple
I set the model executable in config.yaml using the ksh executable listed here for the atmosphere, ocean and ice models. Is there a different coupler executable? I couldn’t find this info in the ksh scripts.
Set other config.yaml variables like jobname, calendar: start: year:
This experiment warm-starts from an existing experiment, so I set the details of that in warm-start.sh (year 500 of PI-GWL-t6)
This particular experiment had land-use change enabled. The ksh scripts inject the new land-use map into the restart file at the end of the year, ready for the next year when the job is resubmitted. payu, however, does this before the start of the year in scripts/pre.sh. I had already injected the new land-use map into the restart file, therefore I needed to modify the script to not do this in the first year. For exemple see here.
This step was specific to my experiment
Point the warm-start-csrio.sh script to my custom restart file.
Remove references to HISTORY. Some of the config files referenced a paths to HISTORY, which is not used by payu. I searched for these with grep and removed them.
Add the ancil files to atmosphere/um_env.py. The ksh scripts conditionally set these based on the time period (I think…). In my case, most of these were the pre-industrial ancil files. There are some ancil files in um_env.py that I couldn’t find in the ksh scripts. Like ARCLBIOG? I ignored them
Run a test and compare the log files to the original version. I used work/atmosphere/atm.fort6.pe0 compared to the equivalent Running.dir/ATM_RUNDIR/um_out/<EXPNAME>.fort6.pe0.
To compare results, I converted the output to netcdf with ACCESS-Archiver and did a cdo sub so I can see where the differences are. This takes a while to do. Is there any way to compare the UM binary output files directly?
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
3
Just so others know and find the correct repo, all the CLEX CMS ACCESS-ESM1.5 experiments are in branches in this repo:
My recollection was that the payu versions did reproduce the original experiments, but my memory is terrible and I wasn’t directly involved in most of the verification. @holger and/or @Scott may be better placed to say for sure.
They would also be well placed to comment on the UM configuration.
Checking bitwise reproducibility is possible just by comparing the checksums in the UM logs. There is comment here from @MartinDix about this
which also mentions mule-cumf , which maybe answers your other question about comparing fields?
If you want ACCESS-NRI assistance with this add a help tag to the post and it’ll be picked up by the triage team.
I’m probably wrong on the other payu versions of standard simulations…
Howerver, no need for help anymore, I have reproduced the results with a payu configuration for my own, but I had quite a bit of confusion along the way. The namelist files are messy. Many of the namelist files are duplicates, so I think they should be cleaned up one day. Only the atmoshpere/namelists file was used as far as I could tell (and the cable one).
The ancillary files were also confusing. It seems payu spams symbolic links in the the run directory which made it hard to know what was being used. Sometimes it would use the ancil file I needed, other times not.
I had trouble using mule-cumf. It said the files from payu and the scripts were not comparable (maybe some difference in header info?), despite there being no numerical difference.
This wasn’t as stright-forward as I had hoped, but I will try doing it again in the near future.
Cheers,
Tam
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
5
Great! (that you managed to reproduce results)
Is this just for your case, or do the standard configs also contain spurious namelists?
I have more than a passing interest, as ACCESS-ESM1.5 is the next model for which ACCESS-NRI will be releasing supported configurations. So if you had any specific suggestions we’d love to hear them.
By design payu does not copy input files (ancils) or restarts, it makes symbolic links to them in the run directory. There are a number of reasons for this approach. A few that come to mind:
easily identify (and clean-up post run) inputs that do not need to be archived
separate configuration and “input”: good from a logical/design point of view, but also facilitates storing configuration, usually text files, in a version control system (git)
This is also very pertinent to us, as we’re going to need to do something similar. Do you have any more information about this? e.g. example files and/or command output?
Is this just for your case, or do the standard configs also contain spurious namelists?
I started with a standard payu configuration and I didn’t add any new namelist files in the atmosphere subdirectory. So I guess some of them are redundant. I didn’t test which ones.
This is also very pertinent to us, as we’re going to need to do something similar. Do you have any more information about this? e.g. example files and/or command output?
Sorry, I must have made some sort of mistake the first time I tried this. mule-cumf works well.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/stashmaster.py:259: UserWarning:
Unable to load STASHmaster from version string, path does not exist
Path: $UMDIR/vn7.3/ctldata/STASHmaster/STASHmaster_A
Please check that the value of mule.stashmaster.STASHMASTER_PATH_PATTERN is correct for your site/configuration
warnings.warn(msg)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
File 1: /scratch/p66/tfl561/access-esm/archive/GWL-NoCrops-B2030/output001/atmosphere/aiihca.pae0dec
File 2: /scratch/p66/tfl561/archive/GWL-NoCrops-B2030/history/atm/GWL-NoCrops-B2030.pa-0500012001
Files compare
* 0 differences in fixed_length_header (with 7 ignored indices)
* 0 field differences, of which 0 are in data
Compared 3345/3345 fields, with 3345 matches
They way I originally compared them:
import numpy as np
import netCDF4 as nc
ncfile_a = nc.Dataset('/g/data/p66/tfl561/ACCESS-ESM/GWL-NoCrops-B2030_payu_test/history/atm/netCDF/GWL-NoCrops-B2030.pa-050012_mon.nc', 'r')
ncfile_b = nc.Dataset('/g/data/p66/tfl561/ACCESS-ESM/GWL-NoCrops-B2030_csiro_test/history/atm/netCDF/GWL-NoCrops-B2030.pa-050012_mon.nc', 'r')
for var in ncfile_a.variables.keys():
difference = ncfile_a.variables[var][:] - ncfile_b.variables[var][:]
print(var, np.any(difference))