Before I make a mistake I want to double check the correct workflow when changing some parameters/flags in the atmosphere/namelist and um_env.yaml during a run.
In the experiment I am running I use no carbon emissions for 50 years, after which I switch on carbon emissions. After 50 years of run, I am thus modifying the atmosphere/namelist and um_env.yaml files. If I do not call “payu setup” after making these changes, will the change of flag (L_CO2-emits= false to true) be taken into account?
Last I did a “payu sweep” and “payu setup” in the middle of a run, it erased my archive directory, maybe I had made a mistake, but I am not so keen to just try it out.
If I do not call “payu setup” after making these changes, will the change of flag (L_CO2-emits= false to true) be taken into account?
Hi @LaurieM,
the updated namelist settings will be picked up automatically without running payu setup. However manually running payu setup when modifying the configuration can help catch simple errors and should be harmless. When adding new input files in the config.yaml, an extra payu setup may also be necessary to ensure the correct PBS storage flags are used.
Last I did a “payu sweep” and “payu setup” in the middle of a run, it erased my archive directory, maybe I had made a mistake, but I am not so keen to just try it out.
The normal payu sweep command should leave the archive directory intact, though payu sweep --hard will delete it. Do you know if you might have run this second command? There has also been a bug in older payu versions prior to 1.1.6 which could cause it to use different archive directories on the login node and when running the model. Do you know which version of payu you were using when this happened?
Thank you Spencer. The issue occurred about 2 months ago, so it was probably either payu 1.1.7 or 1.1.6. It is probably not worth investigating unless the issue occurs again.
The issue I had with payu sweep might be linked to some setup of the experiments as I noted that for some experiments the outputs and restart appear in my home directory.
What I don’t understand is that for the last 5 experiments I run I used the same workflow, but for the 3 first all outputs and restarts only appear in /scratch/archive (with /home/exp/archive being empty) whereas for the 2 latest the outputs and restarts appear in both /scratch/archive and /home/exp/archive. The payu sweep issue appeared when the outputs were also in /home/exp/archive.
The way I set up experiments is that I create a directory in /home/access-esm, copy the relevant files from another experiment without copying the archive directory or other “linked” files; make appropriate modifications, run payu sweep, payu setup. This creates an experiment folder in archive that is the name of my experiment + some random letters and nb. I then copy my restart files in that archive/expt, and re-run a payu sweep and payu setup.
This is not a big nor pressing issue, and I can certainly work around it, but I just wanted to flag it.
Hi Laurie, the home/exp/archive directories should normally just be symlinks to the archive directories on scratch. It sounds like something strange has happened in this case. For each of the three empty /home/exp/archive directories, would you be able to run
Hi @LaurieM, thanks for providing the details on your workflow for creating experiments.
I’ve discussed with the Payu team about the best workflow to use here. In general it’s safest to avoid directly copying files/directories to create new experiments. It can lead payu into an unexpected state, resulting in issues like the the missing archive symlinks described above.
A recommended alternative would be to use the payu clone command to create a new experiment based on a previous one. Under the hood, this uses git to clone an experiment, and it ensures that all the files and settings required by Payu are properly set up.
If you have a template experiment in a directory, e.g. expt-template that you want to produce a new experiment from, you can do the following:
The payu clone command will only copy over changes that have already been committed in the expt-template directory, so the first step is to make sure that all desired changes have been committed. You can check for uncommitted changes with git status. E.g. if I have modified atmosphere/namelists but haven’t committed the changes, git status will show:
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: atmosphere/namelists
git diff can then show the specific line changes that haven’t yet been committed. You can then add and commit the desired changes.
Use payu clone to create a new experiment based on the old one. In its simplest form, the command can be run as:
payu clone <path to expt-template> <path to new experiment>
This will create a new control directory and will setup an archive directory on scratch as well as the required symlink. You can also specify a restart directory for the new experiment to use with:
payu clone --restart <path to restart directory> <path to expt-template> <path to new experiment>
Other options are available for more fine grained git control, and are described in the documentation
The new <path to new experiment> experiment will then be properly set up. You can then make any required changes to its settings, and it should be ready to run.
Let me know if you have any questions about this or if you run into any problems. We’d also be interested in helping community members use some of github based workflows for creating and sharing experiments. Some details are available in the 2024 workshop tutorial, but let us know if you would be interested to learn more.