Thanks @anton
Just noting down some quick updates:
@Harshula has put together a working spack config in PR-10 which uses the ACCESS-ESM1.5 version of MOM5, and the Cable 3 development version of the UM.
@Jhan has been able to run a month without any crashes, however first had to update the cable.nml file in the payu configuration as the ESM1.5 input file pftlookup_csiro_v16_17tiles_wtlnds.csv isnāt suitable for CABLE3.
I think launching CRUNs goes down a different path as there is no um output access.err, wall time used looks like it was 20 minutes but I have no idea what it was doing as work/ was never (re-)created
Occured to me earlier - CABLE3 should work with generic tracers branch. We are running this combination. The LSM and the ocean never come into contact, its all through the coupler
Hi @Jhan,
To continue a payu simulation, you can navigate to the simulationās control directory (that contains the config.yaml file) and use
payu run
This will set off a new run from where the last one finished, and will also run for the length specified under runtime in the config.yaml file.
To set off several (say 10) subsequent runs, you can use
payu run -n 10
and payu will handle the resubmission for each run. More details on this are included here.
In your situation this will be slightly more complicated, as a one month simulation has already been completed. CICE is configured to write a restart file only at the end of each year, and so the simulation will be missing a valid ice restart file to continue from.
If you would like to run in one-month segments, you can edit the CICE namelist so that it writes monthly restarts. To do this, swap dumpfreq = 'y' to dumpfreq = 'm' in the ice/cice_in.nml file.
Youāll then need to restart the simulation from the beginning, as itās missing the ice restart file. One option is to clear the archive with
payu sweep --hard
(Note that this will delete all of an experiments output and restart files, and should be used with care)
And then you can run any number of month long segments with:
payu run -n N
A second option, if you prefer not to delete the existing data is to start a new experiment based on your current control-directory:
First commit any changes to the configuration files, and run
payu clone <path to old control directory> <path to new control directory>
add the CICE dumpfreq changes in the new control directory, and run from there.
Let me know if you have any questions!
very comprehensive thanks @spencerwong
I think this may have just brought up a bug ā we wouldnāt expect it to be able to continue a run at all without the CICE dumpfreq changeā¦ Iāll look into this.
At the end of a successful run, payu will move the access.err and access.out files into the latest archive/outputXYZ directory. They should only remain in the control directory if a run has failed for any reason. Did any of these files end up in your latest output directory?
Thatās quite confusing about the work directory! Usually it should be linked while the simulation is running and then removed at the end. If you look in the latest pre-industrial.o... output file, it should provide some hints as to what directory was used. E.g. in one of my simulations I have:
...
mpirun -wdir /scratch/tm70/sw6175/access-esm/work/cice-cycling-test-c8387c86/ ...
@spencerwong unfortunately I had already swept hard and started a new job before I saw this. I have a clone going as well as I will need to be doing this shortly