Payu: a workflow manager for some ACCESS models

ACCESS-NRI Releases
, , ,
1

What is this?

A topic that will be used to announce updates to payu at NCI.

What is payu?

payu is the tool used to run a number of ACCESS models on NCI hardware

How is this topic used?

This topic will be the place where updates to payu at NCI will be announced, with details of what those updates entail.

What should I do to be notified of any updates?

You can watch this topic, and be notified by email of every update.

Where do I ask questions about releases?

Replies to this topic are disabled.

If you have specific questions about this release follow the guidelines for requesting help from ACCESS-NRI.

If you have questions about payu create a topic in a category that best matches the model you are using, or in the Technical category and tag it with payu. If you require assistance follow the guidelines for requesting help from ACCESS-NRI.

2

17/08/2023

payu has been updated to v1.0.29 in the conda/analysis3-unstable conda environment maintained by the CLEX CMS team.

Updates:

  • Added support for module use to add module paths in config.yaml Issue #347
  • Removed redundant code intended to support MVAPICH Issue #350
  • Supports models built with spack Issue #341
  • Fixed bug introduced with spack support update (version 1.0.28) and cleaned up internal code logic Issue #354

Notes

Enhanced module support

payu already supported loading user-specified modules when a model is run.

payu now also supports module use paths in the config.yaml file. This is documented but in essence if you have an existing config.yaml that has something like this:

modules:
  - netcdf-c-4.9.0
  - parallel-netcdf-1.12.3
  - xerces-c-3.2.3

that only works if you you first do

module use /path/to/module/directory

then the module use is no longer required, it can be incorporated in the config.yaml by changing to:

modules:
   use:
      - /path/to/module/directory
   load:
      - netcdf-c-4.9.0
      - parallel-netcdf-1.12.3
      - xerces-c-3.2.3

This has a couple of benefits:

  1. It makes the configuration more portable. Someone else can clone your configuration and run it without additional steps required
  2. The paths specified in the use section are parsed for storage points and /g/data/ and /scratch directories are automatically added to -l storage options when payu submits the job to the PBS queue

Note: the previous syntax is still supported.

Support for spack built exectutables

For the most part spack uses a mechanism to embed within an executable information about the location of library dependencies that were used to compile the executable.

In these cases it doesn’t rely on some of the module inspection payu does to make sure the correct modules are loaded. For libraries outside the /apps hierarchy payu no longer attempts to do this sort of inspection.

Credits

This work was all done by @jo-basevi. Thanks!

1 Like
3