# Payu: a workflow manager for some ACCESS models **URL:** \ **Category:** ACCESS-NRI Releases\ **Tags:** nci, release, payu, nri-updates\ **Created:** [17 August 2023 06:07 UTC](https://forum.access-hive.org.au/t/payu-a-workflow-manager-for-some-access-models/1098 "2023-08-17T06:07:22Z")\ **Posts on this page:** 1\ **Showing post:** 4
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [16 February 2024 06:53 UTC](https://forum.access-hive.org.au/t/payu-a-workflow-manager-for-some-access-models/1098/4 "2024-02-16T06:53:17Z")
# 18/02/2024 **NOTE** : ACCESS-NRI is now supporting `payu` in a dedicated `conda` environment in the `vk83` project. `vk83` is the project ACCESS-NRI will be using to release all climate models in the future. See [below for more information](#how-to-access-8). This release adds support for experiment UUIDs and marks a major change in the way `payu` names the `work` and `archive` laboratory directories to support experiment UUIDs and `git` branching. The naming scheme incorporates a portion of the experiment UUID and the `git` branch name which prevents namespace clashes, and allows experiments with the same name to co-exist. More importantly it means `git` branches can be utilised seamlessly as independent experiments. As a result the minor version has been incremented (from 1.0 to 1.1): the changes are backward compatible, but from now on by default new experiments will use the new naming scheme. > **[Release Experiment IDs and Branching · payu-org/payu](https://github.com/payu-org/payu/releases/tag/1.1)** > > Notable changes > This release adds support for Experiment UUIDs and marks a major change in the way payu names the work and archive laboratory directories to support Experiment IDs and git branching... ## Updates: - Automatic generation of unique Experiment IDs ([UUIDv4](https://en.wikipedia.org/wiki/Universally_unique_identifier)) - Automatic creation and population of `metadata.yaml` file (compatible with [ACCESS-NRI Intake Catalogue](https://access-nri-intake-catalog.readthedocs.io/en/latest/)) - Uniquely named `work` and `archive` directories, allows the same experiment repository to be used for multiple unique experiments in separate branches - New `payu checkout` command: wraps `git checkout` to facilitate changing between experiments stored in separate branches - New `payu branch` command: lists available branches, and their experiments, that exist in the repository - New`payu clone` command: wrapper for `git clone` that updates metadata and makes sure the experiment directory is correctly configured - New `sync` support: uses `rsync` to copy outputs and restarts to a specified location (can be a local long-term storage disk, or a completely remote machine). Incredibly useful functionality when model outputs are saved to short-term storage, e.g. `/scratch` at NCI - Date based restart pruning: `payu` now supports pandas style date/frequency syntax to specify what restarts should be retained ## Notes ### Experiment UUIDs `payu` now automatically generates unique experiment UUIDs. These are [UUIDv4](https://en.wikipedia.org/wiki/Universally_unique_identifier) format, which is typically represented as a 128bit hexadecimal number, e.g. ```auto 550e8400-e29b-41d4-a716-446655440000 ``` They are guaranteed to be unique within any reasonable computation effort, and so can be confidently used to identify and track experiments. UUIDs are not human friendly, and are designed to be used by software, but the first 8 digits of the experiment UUID is used to uniquely name experiment laboratory `archive` and `work` directories. ### Branches `git` branches are now explicitly supported by `payu`, and form a crucial part of the updated workflow. This means a single `control` directory (which is a `git` repository) can contain multiple independent experiments, and it is possible to switch between experiments, though only one experiment can be active and running at any one time. See the [payu tutorial for more information on branches](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#branches-4). ### Experiment naming An experiment name is used to identify the experiment inside the `work` and `archive` sub-directories inside the _laboratory_. The experiment name historically would default to the name of the _control directory_. This is still supported for experiments with pre-existing archived outputs. To support git branches and ensure uniqueness in shared archives, the new default behaviour is to add the branch name and a short version of the experiment UUID to the name of the _control directory_ when creating experiment names. See the [payu tutorial for more detail](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#experiment-naming-6). ### Syncing `payu` now supports syncing of an experiment archive to another filesystem, either local or remote. There are a number of [configuration options](https://payu.readthedocs.io/en/latest/config.html#postprocessing) to customise what is sync’ed and when. See the [payu tutorial for more detail](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#branches-4). ### Restart pruning `payu` now supports specifying which restarts to retain using date-based frequencies. This allows restarts pruning based on time units. For example setting ```yaml restart_freq: 5YS ``` will only save the first restart of every fifth year, with the rest deleted. See the [payu tutorial for more detail](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#restart-pruning-9). ### How to access payu All ACCESS-NRI models and critical supporting software such as `payu` is located in `/g/data/vk83` on NCI. It is necessary to be a member of the [`vk83`](https://my.nci.org.au/mancini/project/vk83) project to use ACCESS-NRI supported versions of `payu`. See NCI Documentation for more information about [how to join a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project) To access `payu` version 1.1: ```auto module use /g/data/vk83/modules module load payu/1.1 ``` `payu` is installed on `gadi` using an [automated deployment process](https://github.com/ACCESS-NRI/payu-condaenv). This ensures a consistent software environment and the design allows for multiple versions of `payu` to be maintained, so if there are changes which are incompatible with your experiment you can use an older compatible `payu` version. ### Credits All `payu` development was done by @jo-basevi. Deployment by @TommyGatti with assistance from @jo-basevi and @harshula. Thanks! --- _[View the full topic](https://forum.access-hive.org.au/t/payu-a-workflow-manager-for-some-access-models/1098)._