Thanks @spencerwong for reopening this topic!
A similar crash with errorcode 0 just occurred, except it was in the 1st year this time (1850):
MPI_ABORT was invoked on rank 382 in communicator MPI_COMM_WORLD
with errorcode 0.
I rsync’d the control, work, and archive directories in
/scratch/public/bp3051/crash2/
├── archive
├── control
└── work
Let me know if you need any other info!
Hi Benoit, thanks for sharing the files from the simulation. I’ve run into the same error twice just in the last couple of days with my own simulations, and we’re currently trying to better understand the cause.
In all of the cases, the error appears to come from the CICE submodel, which currently doesn’t produce much error information when it crashes. We’re looking into copying over some changes from CM2 which improved the error reporting in CICE, and from there we’re hoping to gather more information on this type of crash if/when it reoccurs.
Cheers,
Spencer
Hi @spencerwong,
Reporting a different crash that just occurred. Although the errors are different (not related to missing restart files AFAICT), I thought replying here may be useful as it is for the exact same configuration (except the age restart file).
I’m not sure how to interpret the errors this time. There are 139 “error” occurrences in access.out, which point to issues with energy continuity and also to a bunch of tracers, at varied locations and globally. In case you want to look at it, I am copying all the files in /scratch/public/bp3051/crash3/. I hope this is helpful! Let me know if I can add any information.
Cheers,
Benoit