I recently set up a piControl run with ESM1.6. I did not initially enable payu sync in the config.yaml. After it had run ~140 years the run stopped, so I tried to run payu sync myself to ensure it was backed up to the right location. To do so, I first updated the config.yaml to include these sync options:
I then ran payu sync from the command line while inside the run directory (run directory is here: /g/data/p66/ajn563/ACCESS-ESM/ESM1.6/piControl-Aus-Exps/. Experiment name: piControl-Aus-Exps-piControl-Aus-Exps-71673a76). I expected payu sync to create a sub-directory inside the base path that would be called piControl-Aus-Exps-piControl-Aus-Exps-71673a76. Instead, it dumped the outputs and restarts into the bath path directly. This means I’ll need a different base path for every experiment I create off this run directory.
If you intend on restarting the run to do more than 140 years you should just be able to rely on making the changes in the config.yaml and payu sync will pick it all up
Base directory to sync archive outputs to, when path is not set. Payu will add the experiment name to the path, where experiment name is the name used for the experiment work and archive directories. Full destination directory is <base_path>/<expt_name>/.
path
Destination path to sync archive outputs to. This must be a unique absolute path for your experiment, otherwise, outputs will be overwritten.
We’re transitioning all the supported configs to have base_path instead of path.
Ah yes, that is probably it thanks @anton . The config.yaml in my config only listed an option for path, not base_path, so I wasn’t aware of that option. Once Gadi is back up tomorrow I’ll give that a try, thanks
@anton one quick follow up. If I only set enable: True and include the base_path in the config.yaml mid-way through the run (e.g. for my run it’s already run 140 years), when I restart the run will payu run go back and sync the first 140 years or only sync the new years?
We think that it will sync the first 140 years again. Sync doesn’t (by default) remove anything from /scratch, and uses rsyncunder the hood. So rsync should see that the files are not there and be able to copy them again from /scratch . We’re not 100% on that and can’t test today as it’s maintenance day, so let us know how it goes!
I’m 99% sure I have done this. Just DONT set a wall time for the sync and leave it at the default 10 hrs. It still will catch up and sync everything eventually provided you are running for an additional length that will allow it. 140 years worth of output though - it might need a few goes anyway
Hi Anton, just following up here. Even though I restarted the run from year 140 with the new payu sync config, the sync did indeed go back and copy all of the files over.
I believe payu doesn’t sync the latest run if the configuration has a postscript (which would be the netCDF conversion here). If the netCDF conversion has successfully finished, I think you can manually sync the final outputs by running payu sync from the control directory
The post-processing did finish i.e. conversion to netCDF was successful for that final output year. I was able to manually sync it no problem but it just seemed a bit odd that it wasn’t handled automatically. It makes sense given the order of these things, but it would be nice if it could take care of that pesky final year for me.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
15