NCI Project(s) Management Tips and Tricks

Technical
1

I have been managing dx2 (and other datasets) for a number of years and I have a few tips to share. I have been putting this off because I wanted it to be comprehensive, but that just means it never gets done.

Managing an NCI project (or multiple projects

There are three aspects to the management of an NCI project: compute, disk (and tape), users (people).

Compute

nci_account is your friend. nci_account -v even more so. This utility lists the users and the amount of compute that they have consumed. This is essential if you want to find where all you allocation is going and to catch a user that might be ‘accidentally’ consuming more than you expected. (‘accidentally’ because there are numerous reasons for this to happen - Wrong queue, too many CPUs, forgotten (or lost) job, wrong project - more on that later). It would be wise to have a regular job that writes this report to a dated file once a week.

There is a helpful tab on mancini called ‘Accounting’. This gives a comparison of current compute usage against allocation. It is an instantaneous calculation and you could do it in your head from the nci_account figures, but it can be helpful in messaging.

Disk (and Tape)

nci-files-report is your friend. nci-files-report --project {project} even more so. This lists all the files under your /g/data/{project} directory that you are using. You can also add nci-files-report --project {project} --filesystem gdata if you want to distinguish gdata over scratch. This report will identify all the users and the disk space that they are consuming. It will also tell you about the users that have files under your project directory, but that are not in the project group. Files that are not in the group mess with quota allocation and will cause you problems (more later).

The alternate report is nci-files-report --group {project} --filesystem gdata and this is valuable in determining who stole your allocation. These are files under another project’s /g/data directory that you cannot see, but are using your allocation. You can identify the ‘offending’ user this way. This is the major reason that the du (disk usage) sum is different to the project report. The other reason is the report is run periodically and there is a delay in reporting.

There are few tools to manage tape (MDSS), but mdss ls -l will list the size of all the files under a directory, which is very helpful.

In summary project tells you the usage under the project and group tells you where your quota went.

Users (People or Wetware)

This is the hardest part.

Who are they?

There is a Linux command, getent passwd {user}, this will allow you to get the name of the user associated with the userid. You can also look them up on mancini.

Communicating with users

There is an envelop icon on mancini that you are able to send messages to all users. This can be sub-optimal as you don’t get read receipts, but as a first point of contact it is excellent. All users in your project have an email against their account and you can get this through mancini. I have in the past scripted a scrape of these emails for more direct interaction.

Why are they in the project

Obviously, most users will be in your project because you approved them, but not always. You may be taking over a project that you are unfamiliar with. From a management point of view, knowing why a user is on the project is pretty important, especially if they are using resources. Some people have joined the project to be able to access the data on the project and they are not expected to be using the compute or disk resources, so if they ‘accidentally’ do use the resources, you need to be able to respond.

Are they still on the project?

People will leave the project and you should remove their access. This is probably the most tedious part of managing the project. You usually cannot contact them and get them to remove files, and possibly you want to keep the files. Emails to NCI Help are really the only way to address this and the simplest message you can send is that the user has left the project and you wish their files to be allocated to your user account. If the user has files in another project, but under your project, same message applies with a please allocate files and directories to the containing project.

Periodic review. You do need to ask users to validate their continued use of the project (periodically). This is where a spreadsheet can help.

Contact me

I am very happy to share code snippets and advice. Please feel free to discuss this with me online and offline.

2

@navidcy, @bethanwhite and @Aidan - FYI and review

Inspired by Difference between nci-files-report and du

3

Wonderful, thank you Griff :heart:
Also tagging @Paul.Gregory who is building a monitoring platform for the CoE projects.

4

If you are a Lead CI, Delegated Lead CI or Chief investigator you can find the registered email for a user in your project by going to https://my.nci.org.au/mancini/project/ab1/members/active (replace ab1 with project code) and clicking on a user name.

5

Yes - This was the way I collected the username to email mapping by scraping these pages - With about 100 users to get, this would have been overly tedious - I think I got your help with the wget and user credentials to do this

6

The link in mancini goes to a URL like https://my.nci.org.au/mancini/project/dx2/member/99999 for each username (abc123) and you can get the list of users using https://my.nci.org.au/mancini/_search/lookup/project_membership/dx2?page=1 (where you iterate through the page numbers) - You then have to filter the active users - A bit fiddly, but quicker that manually clicking through (for large numbers).

7

Nice summary! @griff and thank you for the add-ons @Scott @bethanwhite . Now I know who are managing their projects:) Here’s my two cents.

It is noteworthy that nci_account serves not only as a compute dashboard but also as the primary dashboard for your project within the shell environment. It provides a comprehensive summary of compute, cloud (experimental), and storage usage. The verbose version offers detailed information on compute and cloud usage per user and both compute and storage grants support distribution across stakeholders.

It is the most current record of the project, unlike nci-files-report, which can be 24 hours old.

8

A ‘gotcha’ that I recently encountered involved people using the ARE graphical interface to delete files. The files are not actually deleted, but instead get moved into hidden folders in the base directory for the project. We ended up with a collection of .TRASH* folders in /scratch/jk72/ that were consuming a large proportion of our inode quota.

The permissions on these folders are pretty tight, so I couldn’t access or scan them. The users all claimed that they had deleted lots of files, and scanning their directories showed this reduction, but the project total didn’t go down.

Once we identified the issue, it was straightforward for users to go and delete the folders and files using rm on the command line.

If you’re administering a project, it’s probably worth having a look for similar .TRASH* folders and getting the users to proactively delete them.

9

Some new stuff on this topic

I have recently been concentrating on files that are in one project and are owned by another project. This is a particularly tedious issue and can be difficult to chase down.

I have created 4 functions to help me with this

function mismatched_project_group () {
  nci-files-report --group ${PROJECT} --filesystem gdata --sort-by total_size \
  | awk ‘{if ($3 != $4) print $N}’
}

function mismatched_project_group_users () {
  nci-files-report --group ${PROJECT} --filesystem gdata --sort-by total_size \
  | awk ‘{if ($3 != $4) print $5}’ | grep a-z | sort -u
}

function mismatched_project_group_users_names () {
  for u in $(mismatched_project_group_users) ; do
    getent passwd ${u} | cut -d : -f 5 | cut -d ' ' -f 1 | sed 's/$/,/'
  done | xargs echo
}

function mismatched_project_group_users_emails () {
  mismatched_project_group_users | sed ‘s/$/@nci.org.au/’
}

The function mismatched_project_group will give you a table of the users that have files in the wrong group for the project:

For example:

$ mismatched_project_group
-----------------------BREAKDOWN BY PROJECT, GROUP, AND USER------------------------
FILESYSTEM  SCAN DATE   PROJECT  GROUP  USER         SPACE USED  TOTAL SIZE  COUNT
gdata       2026-04-18  ma14     dx2    jl8309             9.4T        9.4T  224385
gdata       2026-04-24  dp9      dx2    hhz548           841.9G      841.9G  37
gdata       2026-04-09  ce10     dx2    fxw548           808.9G      808.9G  911

The function mismatched_project_group_users will extract the user ids from the report.

The function mismatched_project_group_users_names will use getentto get their first names for an email.

And finally, the function mismatched_project_group_users_emails will append @nci.org.au to the user id so that you can email the offenders directly.

Hopefully, people (Lead CIs) managing projects might find this useful in mamanging their limited resources.

Griff.

10

Thank you Griff!

11

Hi @edoddridge, this is something that we regularly come up against, particularly for researchers whose only interaction with gadi is through ARE rather than using a terminal.

We have to rely on a regular scan and manual request to the users to get them to address the issue.

@navidcy wrote this bash snippet that can be run from a login node for your project of choice, and will list the username and actual name of users who have .Trash directories – that way you can reach out to them directly rather than relying on a mass email that you hope gets read and acted upon by everyone.

NCI are adamant that this is an intended behaviour of .Trash, so there is no recourse there (we have asked!).

for d in /g/data/<project code>/.Trash-*; do
    uid=${d##*.Trash-}
    entry=$(getent passwd "$uid")
    user=$(echo "$entry" | cut -d: -f1)
    gecos=$(echo "$entry" | cut -d: -f5)
    printf "%-8s %-10s %s\n" "$uid" "$user" "$gecos"
done

The output looks like:

<.TrashID> <username> <realname>
12345    aa0000    Firstname1 Lastname1
12346    bb1111    Firstname2 Lastname2
12347    cc2222    Firstname3 Lastname3
12

Thanks for sharing these very useful tips, Griff!

Is there a resource with instructions on when and how to use massdata? All I’ve been able to find online is information from ARCCSS and CLEX!

I need to start managing storage in eg3 more actively, and I think it would be helpful to share some current guidance with project members.

13

NCI Documentation as at 20260928:

The best documentation for MDSS commands is man mdss. It is comprehensive and possibly a little TL;DR.

@zoegillett27 - I too would like a little more guidance on using MDSS - I am currently using it to back up important datasets that need to be kept for archiving (legal) purposes - I believe there are quotas

14

There was interest in the mdssdiff tool I wrote when I was with the old CoE. I provided instructions for installing it in a virtual environment:

But also some folks asked NCI for similar functionality. It is worth adding your voice to that request. The more people who ask the higher the priority.

Note: mdssdiff is a limited tool, but it is a lot better than nothing.

15

Reference: GitHub - coecms/mdssdiff: Look for differences between local filesystem and it's copy on a mass data store system · GitHub

Description: Provides a comprehensive wrapper that can compare files (recursively) in a local directory with the files on MDSS. Differences in file size and modified time are reported, including differences in contents of directories.

16

Thanks to Scott Wales for filling the gaps in my knowledge.

I have been able to install the mdssdiff utility that Aidan Heerdegen (and others) wrote many years ago on NCI (gadi)

There were several fairly simple steps to this process and all of the relatively painless.

  1. Get a good version of python

module use /g/data/xp65/public/modules/

module load conda/analysis3

Confirm for yourself that this worked with which python.

  1. Create a virtual environment for mdssdiff (or whatever suits, could be bob or tools)

python -m venv --system-site-packages mdssdiff

The -m is so that you use the xp65 version of python’s venv command.

The --system-site-packages means that you are using the xp65 version of python and not installing a new (or another) one.

  1. Activate the virtual environment

source mdssdiff/bin/activate

  1. Get the mdssdiff repository

git clone https://github.com/coecms/mdssdiff.git

cd mdssdiff

  1. Pip (pip) install this in your venv (using the python module version -m)

python -m pip install .

  1. Check it works

mdssdiff --help

Again, thanks to Scott Wales for filling the gaps and improving my understanding and Aidan for writing it.