I have been running a freshwater hosing experiment with v1.6 of the ESM and it is now consistently crashing after 272 years of integration due to a segfault, apparently originating somewhere in the atmosphere model.
That process was terminated (SIGTERM). The PBS job may have run out of memory or walltime (check the footer) or perhaps one of the other ranks has a SIGSEGV causing all the ranks to terminate, see if you can find a trace for that rank.
Echoing scott’s comments - process killed usually means the job has run out of time or memory and is killed by the scheduler. Usually this shows up in the PBS logs. Can you find them and report back?
Hi Willem,
Your home directory doesn’t have group read permissions. Given that I’m in your group (y99), you could add them for someone like myself to check over by doing: chmod -R g+rX /home/561/wnh561
Thanks for the quick replies. @dkhutch the directory should be accessible now. Sorry, should have included the footer from the error log. It shows:
prterun noticed that process rank 163 with PID 1796869 on node gadi-cpu-spr-0570 exited onsignal 11 (Segmentation fault).
And the backtrace implies something going wrong in the atmosphere (though I’m not really that familiar with ACCESS). There is much more than what is shown below; I’ve attached the .err and .out files to this post (the .out shows conservation warnings prior to the crash which I didn’t see in previous runs, but not sure if this is relevant).
[gadi-cpu-spr-0570:1796869:0:1796869] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
[gadi-cpu-spr-0570:1796885:0:1796885] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
==== backtrace (tid:1796885) ====
0 0x0000000000012990 __funlockfile() :0
1 0x0000000000a66bac interpolation_() /scratch/tm70/tm70_ci/tmp/restricted/spack-stage/spack-stage-um7-git.2026.04.000_access-esm1.6-bdlpvvfpydgedzzktzviqovdeb3fyzbt/spack-src/ummodel_hg3/ppsrc/atmosphere/dynamics_advection/interpolatio
n.f90:993
Hi @WillemH, the atmosphere log file in /home/561/wnh561/access-esm1.6/rel_pi+em+CN/work/atmosphere/atm.fort6.pe0 finishes with
==============================================
initial Absolute Norm : 67248670.7694857
GCR( 2 ) failed to converge in 50
iterations.
Final Absolute Norm : 1049.99880463915
==============================================
suggesting a failure in the solver. This can be caused by many different reasons, but often points to a random numerical issue. It’s worth trying to add a random perturbation to the last atmospheric restart to see if it gets around the error. We’re currently writing up some updated recommendations on how to do this, but I’ll copy in a draft here for now:
Load the model-processing environment (note, this may clash with payu and so it’s best to do this in a session without payu loaded):
module use /g/data/vk83/modules
module load model-processing
Navigate to the latest restartXYZ directory in the experiment’s archive directory, and rename the atmospheric restart file.
cd restartXYZ/atmosphere
mv restart_dump.astart restart_dump.astart_orig
It will be worth keeping this unperturbed restart as a backup.
Apply a perturbation to the atmospheric restart file
here <SEED> can be any integer, and it’s used to set the random seed for the perturbation. Specifying a random seed is important, as it allows for the exact same perturbation to be reapplied in the future. Make sure to keep track of whichever value you use.
This will apply a random perturbation with amplitude 0.01 to the atmospheric theta field. If you’d like to change the amplitude, you can add a -a <AMPLITUDE> argument.
The resulting restart will be written to restart_dump.astart. We need to use this file name for the model to be able to pick up the restart.
Unload the model-processing environment and load the payu environment:
module unload model-processing
module load payu
Make a record of the perturbation in the experiment runlogs. First cd into the Payu control directory for the experiment and run
payu setup
This will rewrite the manifest file using the data from the modified restart. To record the pertubation in the experiment history, next run
$ git commit -a -m "restartXYZ atmospheric restart perturbed using command: perturbIC -s <SEED> restart_dump.astart_orig -o restart_dump.astart"
filling in the correct information for <SEED>.
If you run the git log command, it should now include the above record.
You will finally need to run payu sweep to clear the work directory before setting off the next run with payu run.
Hopefully the crash will be avoided after applying the perturbation. If it occurs again at the same point, it could suggest an issue in the configuration.
Let me know if you have any questions about those steps or if you run into any issues.
Thanks @spencerwong for this excellent outline of the perturb_IC process with git-logged perturbation history. @WillemH I was also thinking perturb_IC is your best bet to try first.
This is fantastic, thanks for the detailed instructions. I’ve done this and now re-submitting the job. I’ll just note that at step 5 when I run payu setup, it threw an error because the work directory still existed, so I had to sweep first. Will keep you all updated…
Apologies, I didn’t have time to check the experiment yesterday. I’ve run another 30 years with the new atmos. restart without any further error, so I would say that’s a success! Thanks again to everyone for such fast assistance.