I’m running some perturbation experiments with the pre-industrial control in ACCESS-ESM1-5 but since last week a number of them have started crashing during the run and I don’t know why.
I get the error message: “payu: Model exited with error code 139; aborting.”. I can see this has been mentioned in other posts, but I don’t think this is relevant here because I have started these experiments using the PI-02 restarts.
I’d like some help finding and interpreting the errors. An example control directory which has crashed at time step 4525 is here: /home/561/hd4873/PostDoc/ACCESS-ESM/access-esm-payu
The work directory is here: /scratch/e14/hd4873/access-esm/work/access-esm-payu-ocean-warm-upwelling_year780-6213760d
and the error logs are here: /scratch/e14/hd4873/access-esm/archive/access-esm-payu-ocean-warm-upwelling_year780-6213760d.
I know it might sound unlikely, but have you checked if the error occurs again if you simply do a sweep and re-run it?
ACCESS-ESM1.5 has a habit of crashing for unknown reasons, and I find it’s best to first check if the error occurs twice. (Sometimes it just runs fine the second time you try…)
Have you tried perturbing the model using /projects/access/apps/pythonlib/umfile_utils/perturbIC.py. I encountered this error, and I’m not sure why, but perturbing sometimes does the trick.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
8
A crash in the bi_linear_h routine is often the result of the model becoming unstable and if so can fixed with a small perturbation of the atmosphere as @HIMADRI_SAINI suggested.
@Aidan@HIMADRI_SAINI sorry, what’s the syntax for running this? I’ve just tried like so (from the CLEX instructions): /projects/access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart
So I tried copying the script and running perturbIC -arguments file as suggested above (what’s considered a small perturbation by the way?), but no luck there - think I’m getting the syntax wrong.
Hi Hannah,
This seems like a python2 / python3 problem. The perturbIC.py in the example is using the old print method, where print is done without brackets. A couple of ways to deal with this (not sure what Himadri did…):
explicitly call python2 interpreter
copy the script to a local directory and update to print() with brackets.
Probably the first way is easier because the script has dependencies to the umfile.py and um_fileheaders.py script in the same directory.
@dkhutch thanks! I loaded python2/2.1.7 and ran python2 /projects/access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart and I think that’s worked.
3 Likes
MartinDix
(Martin Dix ACCESS-NRI Associate Director for Model Development)
14
There’s also a version updated for python3 available in the pythonlib/umfile_utils/access_cm2 module. It differs in requiring a seed as an argument which makes it reproducible
module use ~access/modules
module load pythonlib/umfile_utils/access_cm2
perturbIC.py -h
usage: perturbIC.py [-h] [-a AMPLITUDE] -s SEED ifile
Perturb UM initial dump
positional arguments:
ifile Input file (modified in place)
options:
-h, --help show this help message and exit
-a AMPLITUDE Amplitude of perturbation
-s SEED Random number seed (must be non-negative integer)
What does this mean for publication purposes though? At the moment I’m testing so it’s fine, but if this had to be done for an ensemble that I wanted to publish - is this perturbing method considered okay by the Earth System community if you use the reproducible method that @MartinDix posted? Or is it not really okay to publish runs that have had to be perturbed in this way?
My feeling is that making a small perturbation during spin up like this is not at all problematic for publication. Plenty of models have to do tricks like this… It is basically a non-issue. (If you want to document instances of perturbations like this then great, that’s probably better than what most people do.)
Dr David Hutchinson (he/him)
ARC DECRA fellow in paleoclimate modelling
Climate Change Research Centre, UNSW Sydney
david.hutchinson@unsw.edu.au
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
17