Fixed bug introduced with spack support update (version 1.0.28) and cleaned up internal code logic Issue #354
Notes
Enhanced module support
payu already supported loading user-specified modules when a model is run.
payu now also supports module use paths in the config.yaml file. This is documented but in essence if you have an existing config.yaml that has something like this:
It makes the configuration more portable. Someone else can clone your configuration and run it without additional steps required
The paths specified in the use section are parsed for storage points and /g/data/ and /scratch directories are automatically added to -l storage options when payu submits the job to the PBS queue
Note: the previous syntax is still supported.
Support for spack built executables
For the most part spack uses a mechanism to embed within an executable information about the location of library dependencies that were used to compile the executable.
In these cases it doesn’t rely on some of the module inspection payu does to make sure the correct modules are loaded. For libraries outside the /apps hierarchy payu no longer attempts to do this sort of inspection.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
4
18/02/2024
NOTE: ACCESS-NRI is now supporting payu in a dedicated conda environment in the vk83 project. vk83 is the project ACCESS-NRI will be using to release all climate models in the future. See below for more information.
This release adds support for experiment UUIDs and marks a major change in the way payu names the work and archive laboratory directories to support experiment UUIDs and git branching. The naming scheme incorporates a portion of the experiment UUID and the git branch name which prevents namespace clashes, and allows experiments with the same name to co-exist. More importantly it means git branches can be utilised seamlessly as independent experiments.
As a result the minor version has been incremented (from 1.0 to 1.1): the changes are backward compatible, but from now on by default new experiments will use the new naming scheme.
Updates:
Automatic generation of unique Experiment IDs (UUIDv4)
Uniquely named work and archive directories, allows the same experiment repository to be used for multiple unique experiments in separate branches
New payu checkout command: wraps git checkout to facilitate changing between experiments stored in separate branches
New payu branch command: lists available branches, and their experiments, that exist in the repository
Newpayu clone command: wrapper for git clone that updates metadata and makes sure the experiment directory is correctly configured
New sync support: uses rsync to copy outputs and restarts to a specified location (can be a local long-term storage disk, or a completely remote machine). Incredibly useful functionality when model outputs are saved to short-term storage, e.g. /scratch at NCI
Date based restart pruning: payu now supports pandas style date/frequency syntax to specify what restarts should be retained
Notes
Experiment UUIDs
payu now automatically generates unique experiment UUIDs. These are UUIDv4 format, which is typically represented as a 128bit hexadecimal number, e.g.
550e8400-e29b-41d4-a716-446655440000
They are guaranteed to be unique within any reasonable computation effort, and so can be confidently used to identify and track experiments. UUIDs are not human friendly, and are designed to be used by software, but the first 8 digits of the experiment UUID is used to uniquely name experiment laboratory archive and work directories.
Branches
git branches are now explicitly supported by payu, and form a crucial part of the updated workflow. This means a single control directory (which is a git repository) can contain multiple independent experiments, and it is possible to switch between experiments, though only one experiment can be active and running at any one time.
An experiment name is used to identify the experiment inside the work and archive sub-directories inside the laboratory.
The experiment name historically would default to the name of the control directory. This is still supported for experiments with pre-existing archived outputs. To support git branches and ensure uniqueness in shared archives, the new default behaviour is to add the branch name and a short version of the experiment UUID to the name of the control directory when creating experiment names.
payu now supports syncing of an experiment archive to another filesystem, either local or remote. There are a number of configuration options to customise what is sync’ed and when. See the payu tutorial for more detail.
Restart pruning
payu now supports specifying which restarts to retain using date-based frequencies. This allows restarts pruning based on time units. For example setting
restart_freq: 5YS
will only save the first restart of every fifth year, with the rest deleted.
All ACCESS-NRI models and critical supporting software such as payu is located in /g/data/vk83 on NCI. It is necessary to be a member of the vk83 project to use ACCESS-NRI supported versions of payu. See NCI Documentation for more information about how to join a project
To access payu version 1.1:
module use /g/data/vk83/modules
module load payu/1.1
payu is installed on gadi using an automated deployment process. This ensures a consistent software environment and the design allows for multiple versions of payu to be maintained, so if there are changes which are incompatible with your experiment you can use an older compatible payu version.
All ACCESS-NRI models and critical supporting software such as payu is located in /g/data/vk83 on NCI. It is necessary to be a member of the vk83 project to use ACCESS-NRI supported versions of payu. See NCI Documentation for more information about how to join a project
To access payu version 1.1.3:
module use /g/data/vk83/modules
module load payu/1.1.3
payu is installed on gadi using an automated deployment process. This ensures a consistent software environment and the design allows for multiple versions of payu to be maintained, so if there are changes which are incompatible with your experiment you can use an older compatible payu version.
This new version has been deployed to vk83 and is now the default version. It contains a number of bug fixes and improvements and is the recommended version to use. The previously deployed version (1.1) is deprecated and will be removed when the next version of payu is deployed, or within 3 months, whichever is sooner.
module use /g/data/vk83/modules
module load payu/1.1.5
Updates
Get module executables from paths added by loaded environment modules (#439, #482) Allow generic tracers with cesm_cmeps driver (#433) Remove unnecessary UM config files (#455) Replace UM um_env.py configuration file with yaml file (#459) User scripts and postscript updates to error handling, environment variables and shell features (#452, #467, #445) Adding date based pruning for ACCESS-ESM1.5 (#465) Changes to manifest logic (#475) Enable seperate ice_history.nml & cice_in.nml settings (#483) Replace CICE start date calculations (#484) Add command line flag to disable metadata generation and related commits (447)
For a full list of pull requests that includes additional bug-fixes, see the Payu Github 1.1.5 release.
Notes
Python version 3.10
Released payu environments modules from payu/1.1.5 onwards are based on python version 3.10.
UM um_env.py files
UM um_env.py configuration files are no longer supported and should be replaced with um_env.yaml files. Existing um_env.py files can be converted to yaml files, using this script.
Loading model executables using model modules
Payu can now find model executables by searching paths added to $PATH by model environment modules. This simplifies the config.yaml, as only the name of the executable is required and changing model versions is simpler and less error prone. For example,
Previously, the executable had to be specified as a full path. For executables built by spack, these included a hash in the path, and were long and complicated. For example:
Loaded modules in config.yaml must be unique to ensure the correct model executables is used. This means modules must be specified with a version, and modules of the same name and version can not be found in multiple module directories.
Updates to user processing scripts
Payu now exports some current run information to environment variables so they can be accessed in post-processing scripts:
PAYU_CURRENT_RUN - The current run number, e.g. 0 for first run, 1 for the second run
PAYU_ARCHIVE_DIR - Full path to the archive directory - this contains all the outputs and restarts subdirectories
PAYU_CURRENT_OUTPUT_DIR, PAYU_CURRENT_RESTART_DIR - Full path to the current output and restart directories, e.g. for first run, it would be /path/to/archive/output000 and path/to/archive/restart000
Userscript and postscript commands calls can now also include shell-specific values such as file re-directions, pipes and environment variables (which are expanded). So it’s now possible to run commands such as:
If users scrips exit with an error payu run execution halts. If this is not desirable, error handling will need to be added to post-processing scripts. Previously only warnings were issued if a user scripts exited with an error.
Changes to manifest logic
Payu manifests store information of files in the work directory of an experiment. They are used to track changes to files over an experiment for experiment provenance, and ensure an experiment run can be reproduced. There are three manifests types: executable files, input and restart files (exe.yaml, input.yaml and restart.yaml respectively).
The logic for updating manifests and enforcing reproducibility has been greatly simplified:
Stored manifests from previous runs are used as the source of truth for full (md5) hashes, for all manifest types. Previously this was the case for only input manifests.
Fast change sensitive hashes, by default binhash, are calculated at each payu setup. If a fast hash matches the value in the stored manifest the full hash from the stored manifest is used. This avoids re-calculating slow md5 hashes where possible.
All changes to config.yaml are now correctly detected. For example a different executable path or new input file paths.
scaninputs option has been removed. This allowed existing file paths to change but not scan for new inputs. This means only paths configured in config.yaml or found through searching input and restart directories, are added to the work directory.
Enforcing reproducibility
Setting reproduce to true in config.yaml or via a command-line option will check and make sure files have not changed since the previous run. It is possible to set reproduce for each manifest type separately.
When reproduce for a manifest type is set to true, payu will refuse to run if:
Full hash changes: calculated md5 hashes differ to full hash in stored manifest
New files: if files are found in the work directory that were not in the stored manifest
Missing files: files in the stored manifest are not present in the work directory
If a full path to a file has changed or a fast hash has been changed, but there’s a match with the stored full hash (so it is effectively the same file), the manifest will be updated.
Changes to config.yaml are now correctly picked up. For example a different executable path or new input file paths. Previously specifying reproducibility would only add paths in the manifests to the work directory and raise errors if those files were modified.
Release: 1.1.5
Get module executables from paths added by loaded environment modules (