Workflow when changing atmosphere/namelist um_env.yaml during a run

Earth System Earth System Model
10

Hi @LaurieM, thanks for providing the details on your workflow for creating experiments.

I’ve discussed with the Payu team about the best workflow to use here. In general it’s safest to avoid directly copying files/directories to create new experiments. It can lead payu into an unexpected state, resulting in issues like the the missing archive symlinks described above.

A recommended alternative would be to use the payu clone command to create a new experiment based on a previous one. Under the hood, this uses git to clone an experiment, and it ensures that all the files and settings required by Payu are properly set up.

If you have a template experiment in a directory, e.g. expt-template that you want to produce a new experiment from, you can do the following:

  1. The payu clone command will only copy over changes that have already been committed in the expt-template directory, so the first step is to make sure that all desired changes have been committed. You can check for uncommitted changes with git status. E.g. if I have modified atmosphere/namelists but haven’t committed the changes, git status will show:

    Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   atmosphere/namelists

    git diff can then show the specific line changes that haven’t yet been committed. You can then add and commit the desired changes.

  2. Use payu clone to create a new experiment based on the old one. In its simplest form, the command can be run as:

    payu clone <path to expt-template> <path to new experiment>

    This will create a new control directory and will setup an archive directory on scratch as well as the required symlink. You can also specify a restart directory for the new experiment to use with:

    payu clone --restart <path to restart directory> <path to expt-template> <path to new experiment>

    Other options are available for more fine grained git control, and are described in the documentation

  3. The new <path to new experiment> experiment will then be properly set up. You can then make any required changes to its settings, and it should be ready to run.

Let me know if you have any questions about this or if you run into any problems. We’d also be interested in helping community members use some of github based workflows for creating and sharing experiments. Some details are available in the 2024 workshop tutorial, but let us know if you would be interested to learn more.

Cheers,
Spencer