How to preserve reproducibility when applying perturbations in the event of a numerical instability crash

Is there an updated version of
/g/data/access/projects/access/apps/pythonlib/umfile_utils/perturbIC.py
available?

I’m looking to apply perturbations to a restart_dump.astart file for ACCESS-ESM1.5 AMIP experiments. I wanted to check whether the existing (or any updated) version of this script is compatible with ACCESS-ESM1.5 restart dumps, and whether it can be used directly with the following command:

python /g/data/access/projects/access/apps/pythonlib/umfile_utils/perturbIC.py -a 0.05 restart_dump.astart

Have you tried it? Does it work?

NB: please make a copy of the restart file and test on that!

Thanks @Aidan, the script runs successfully after a minor Python 3 syntax update. I’ll only be able to verify whether the output file works once I have some CPU time on NCI.

1 Like

The ACCESS-ESM1.5 AMIP run crashes due to an atmosphere basis time mismatch when using the restart_dump.astart file modified with perturbIC.py. Although xconv indicates that the astart file has the correct date stamp, the model still reports a date mismatch and aborts before reading the ancillary files.

I’ve attached a vimdiff log comparing runs using the original and perturbed restart_dump.astart files for reference. Any insight into what metadata or time information may be altered by perturbIC.pywould be greatly appreciated.

Hi @abhik, the perturbIC.py script shouldn’t be affecting the time information. The atmosphere basis time missmatch crash often comes up when the date in the restart file doesn’t match the date in the um.res.yaml calendar file that gets paired with it, e.g. the atmospheric restart directory for the historical configuration /g/data/vk83/configurations/inputs/access-esm1p5/modern/historical/restart/atmosphere contains

README  restart_dump.astart  um.res.yaml

and the um.res.yaml file contains:

end_date: 1850-01-01 00:00:00

The error can also come up when the um.res.yaml is omitted from the restart directory.

For the failing simulation, does the date in the um.res.yaml file match the restart?

Hi @spencerwong, I wasn’t aware that um.res.yaml might be required in the restart directory, as the original restart file runs correctly from the same location. Nevertheless, I copied and modified um.res.yaml into the restart directory:/g/data/k10/sza565/access-esm/restart/, but this did not resolve the date mismatch issue.

Hi @abhik, I’ve just taken a quick look at the directory. Payu needs the um.res.yaml to be in the same directory as the restart_dump.astart file, would you be able to copy it there and check if the error still occurs?

Hi @spencerwong, both restart_dump.astart_p0.05 and um.res.yaml are in /g/data/k10/sza565/access-esm/restart/. But that doesn’t solve the problem.

Hi @abhik, just to clarify the um.res.yaml file needs to be directly next to the restart_dump.astart file. I think currently the restart_dump.astart file is in the directory /g/data/k10/sza565/access-esm/restart/restart_1995_p0.05 whereas the um_res.yaml file is one level up, in /g/data/k10/sza565/access-esm/restart. Could you try copying the um_res.yaml file into /g/data/k10/sza565/access-esm/restart/restart_1995_p0.05?

Hi @spencerwong, thanks for pointing this out. The model is now running with the modified restart_dump.astart without any issues.

2 Likes