Adding data to the COSIMA community WG space `ol01`

COSIMA Working Group
, ,
1

This post is to describe how the NCI COSIMA community WG space ol01 is managed. The approach is meant to be light weight and the important thing is that anyone that has a dataset that is of interest to others (e.g. 4 people or more), can readily share it.

The below looks onerous but a few small steps help us avoid (well trodden!) issues down the line. Do ask if anything is unclear.

How do I add my data to ol01?

Please post a reply in this topic and specify: your NCI user name, the full path to the directory you would like created, the dataset size, how long you wish for it to be available and the nci usernames of any other people who need write access to the directory.

Procedure

Going forwards, we plan to manage ol01 via:

  • Using a very small group of people on the ol01_w writer’s group (four or five experienced users) – these few people can read and write to anywhere on the space (provided the below procedure is followed).
  • Requests for a new space are directed to the ol01_w writer’s group (see above in "How do I add my data to ol01? ").
  • When a member of the writer’s group creates a new space for a user:
    • gives write permissions to the ol01_w writer’s group;
    • gives write access to the user requesting the space;
    • gives write access to the list of people that will need write access to that folder (the permissions for the new folder are set by the person from the writer group who creates the directory).

This process ensures that when people go on leave/change roles, it is still possible for us to manage the data.

Useful commands:

How to add the ol01_w group to have write permissions:

setfacl -Rm g:ol01_w:rwX,d:g:ol01_w:rwX,o:r-X,d:o:r-X,m:rwX,d:m:rwX /g/data/ol01/DIRECTORY

How to give a specific person write permissions:

setfacl -Rm u:abc123:rwX,g:ol01_w:rwX,d:g:ol01_w:rwX,o:r-X,d:o:r-X,m:rwX,d:m:rwX /g/data/ol01/DIRECTORY/SUB-FOLDER
1 Like
2

I’d like to share some test runs of OM2+WOMBATlite that are being run as we prepare official releases.

Currently, I have ~2TB of 1 deg and 0.25 deg data that I’d like to share. At some point soon, I’ll also be doing some 0.1 deg runs that will be larger. All these data can be removed once the configuration releases have been made.

I’m a member of ol01_w so I don’t need the directories created. Is it okay if I move these data to ol01?

ADDED:
Suggested locations: /g/data/ol01/outputs/access-om2-1deg[0.25deg,0.1deg]
Write access: no one else should need write access at this point

3

Sounds fine to me. @adele-morrison @aekiss that ok?

4

How long do you think you’ll need this space @dougiesquire ? We have room (10.8 TB free of 100TB total), but will need to start deleting/archiving old runs at some point.

5

I’d imagine only a few weeks. Though I’ve just discovered that the primary person I wanted to share the data with (@pearseb) actually has access to the data’s current location. So let’s park this until it’s actually needed. Apologies for the noise!

1 Like