I had a few runs going over the weekend and had a few crashes.
I had one case of the ocean/wombat error message which I’ve added to floating invalid crash in Wombat in August spinup · Issue #182.
I had another JobFS exceeded error. Seems to happen at the end of the year when it’s trying to copy to the archive directory. Stops when it has tried to use ~10 times more JobFS (~800Mb) than is used in other cases (8.32 Mb). See /g/data/p66/rml599/amip-test/amip-exp6/archive/pbs_logs (with *.keep being a copy of the failed year).
When looking at why some of my jobs had failed, I happened to notice that the atm.fort6.pe207 file was much larger than the equivalent files for other .pe numbers. This appeared to be the same for all cases that I had running and looked like it might be due to output related to a river-routing error (@inh599, @spencerwong). Now my runs have finished, I don’t have the files but this directory (at least while the model is currently running) seems to be showing the same behaviour.
/scratch/p66/jxs599/access-esm/work/AugustSpinUp-Jhan-dev-20250808-1-d1b0b669/atmosphere