Changing project codes for payu control directories under `/g/data`

Technical
, , , ,
1

About

How to run payu experiments from control directories under /g/data with non-matching or multiple compute project codes.

Who is this for?

Anyone whose uses payu on NCI and their payu control directory is located on /g/data, and the compute project code they use to run their experiments does not match the /g/data project code where their control directory is located.

The most common use case is when users exhaust the quota on one project and need to change the compute project code to run the experiment, but keep the control directory in the same location.

This advice does not apply if the

  1. /g/data project where the control directory is located is the same as the compute project code used for running an experiment.
  2. the control directory is in the users $HOME directory (generally recommended)

Ok, what do I need to do?

Run this command on your payu control directory:

chmod --recursive g+s <path to control directory>

and replace <path to control directory> to the location of your control directory under /g/data.

Why is this necessary?

NCI /g/data directories are governed by strict disk quotas that are project based. When writing files into a /g/data directory there can be issues if the files have a group ID belonging to a project that does match the /g/data directory group ID.

When payu submits a run to the PBS queue the compute project code is the default group ID of the files that are written during that run. Where compute and storage projects don’t match it can causes issues with disk quotas, resulting in errors about files not being able to be written to the control directory

How does the command work?

The command above sets the setgid permission on the control directory and all its sub-directories.

This ensures that files created by payu inside the control directory inherit that directory’s project, rather than using the specified compute project. This can prevent problems, for example when using a compute project with no /g/data allocation.

You can confirm this has worked using ls -l and checking the group executable bit, it should be set to s, e.g.

$ ls -l
drwxr-s---  7 aaa999 xx00  4096 Feb  2 16:31 access-esm

What if I already ran under different compute projects?

If your control directory is under a /g/data project and you have already used compute projects that are not the same as the project where the control directory is located, you may already have some files with an incorrect group.

You can ensure all files have the same group by running chgrp, e.g.

chgrp --recursive <PROJECT> <path to control directory>

and replace <PROJECT> and <path to control directory> as required.

How can I avoid this problem

If possible keep your payu control directories in a sub-directory of $HOME. This does not have the same problem because disk quotas are calculated differently for that filesystem.