Hi everyone, I have been running a freshwater hosing experiment with v1.6 of the ESM and it is now consistently crashing after 272 years of integration due to a segfault, apparently originating somewhere in the atmosphere model. Error log shows: forrtl: error (78): process killed (SIGTERM) Image …

Hi @WillemH , the atmosphere log file in /home/561/wnh561/access-esm1.6/rel_pi+em+CN/work/atmosphere/atm.fort6.pe0 finishes with ============================================== initial Absolute Norm : 67248670.7694857 GCR( 2 ) failed to converge in …

That process was terminated (SIGTERM). The PBS job may have run out of memory or walltime (check the footer) or perhaps one of the other ranks has a SIGSEGV causing all the ranks to terminate, see if you can find a trace for that rank.

[image] WillemH: forrtl: error (78): process killed (SIGTERM) Hi @WillemH , Echoing scott’s comments - process killed usually means the job has run out of time or memory and is killed by the scheduler. Usually this shows up in the PBS logs. Can you find them and report back? Thanks, Ed

[image] WillemH: home/561/wnh561/access-esm1.6/rel_pi+em+CN Hi Willem, Your home directory doesn’t have group read permissions. Given that I’m in your group (y99), you could add them for someone like myself to check over by doing: chmod -R g+rX /home/561/wnh561

Hi all, Thanks for the quick replies. @dkhutch the directory should be accessible now. Sorry, should have included the footer from the error log. It shows: prterun noticed that process rank 163 with PID 1796869 on node gadi-cpu-spr-0570 exited onsignal 11 (Segmentation fault). And the backtrace i…

Thanks @WillemH Can you also provide a job id?

Hi @edoyango The job ID for the most recent run is Job Id: 178445086.gadi-pbs Though this is the second time it’s crashed in the same way (I restarted to make sure it was consistent).

Thanks @spencerwong for this excellent outline of the perturb_IC process with git-logged perturbation history. @WillemH I was also thinking perturb_IC is your best bet to try first.

Hi @spencerwong This is fantastic, thanks for the detailed instructions. I’ve done this and now re-submitting the job. I’ll just note that at step 5 when I run payu setup, it threw an error because the work directory still existed, so I had to sweep first. Will keep you all updated…

Hi @WillemH , how did the perturb_IC go?

Hi all, Apologies, I didn’t have time to check the experiment yesterday. I’ve run another 30 years with the new atmos. restart without any further error, so I would say that’s a success! Thanks again to everyone for such fast assistance.

ACCESS ESM1.6 experiment segfault

Earth System Earth System Model

Topic		Replies	Views
Need help finding and interpreting ACCESS-ESM1-5 errors Earth System help	18	309	21 August 2024
"Run ACCESS-ESM" fails with error code 139 Earth System Model help , mpi , access-esm , cice4	17	663	8 February 2024
ACCESS-ESM 1.6 job not ending General help , access-esm , access-esm16	10	163	4 August 2026
Restart problem (atmosphere) Earth System help , payu , restart , access-esm	22	798	24 March 2025
Need help: ¼ MOM ACCESS-ESM1-5 coupled run, crashed!?!? Technical help	5	168	24 February 2025

ACCESS ESM1.6 experiment segfault

Related topics