Parcels particle tracking and inode quota on Gadi

COSIMA
, , ,
1

Hi everyone,

We (jk72) have been encountering inode quota issues on Gadi lately.

Our users with the biggest footprints have all been using the parcels particle tracking library, which now outputs .zarr files by default. This means that even relatively modest particle tracking experiments can rack up more than a million files very quickly.

I know it’s possible to combine them into either NetDCF or tarballs after the fact, but is there a better approach?

Efforts to deal with this are also being hampered by the fact that the most recent scan for nci-files-report is nearly three weeks old for /scratch/jk72/. I asked the Helpdesk about this and was told that a rework of the scanning protocol is in progress, but that there is no ETA for an updated scan. The NCI Project(s) Management Tips and Tricks post is really helpful, but without an update to the output for nci-files-report I feel pretty hamstrung.

Thanks,
Ed

2

My zarr knowledge is a little out of date, but you can zip zarr collections and still access them directly with Python - see storage - zarr-python

3

You can run your own scans for usage / inode count using ncdu kindly provided by the release team:

module use /g/data/vk83/modules
module load system-tools/ncdu/1.19-1
ncdu /scratch/tm70/$USER/

Use the help (?) interface to sort by inodes

It takes a while for large folders

4

Definitely only one person do this (so coordinate efforts). Otherwise you thrash the filesystem. @clairecarouge has been in touch NCI about the scanning issue, so definitely good to loop her into discussions about problems you are having.

The ncdu man page is here:

The file count options are explained in the KEYS section.

You can output the scan to a file and then examine later. That may be a good option if you want to share the output with others, or run the job itself under a PBS job (as it can take quite a while).

5

Just to note that we have an open issue on the COSIMA recipes about this also with more discussion from last time this issue arose:

6

For triage reasons, I’ve added the community-help tag here. We know a lot about managing storage on Gadi but do not officially support requests on this question outside the ACCESS-NRI scheme.

You already have gotten a lot of information here. I don’t have more to add at this point.