Difference between nci-files-report and du

Technical
1

Hi team,

I am managing my storage usage on Gadi, and I find a big difference between nci-files-report and du:

nci-files-report --project if69 --filesystem gdata --sort-by total_size gives a space usage of 170.7 G
Screenshot 2025-04-23 at 8.54.09 am

However, I went to my directory under the project and run du -sch /g/data/if69/cj0591. It only returns 50 G
Screenshot 2025-04-23 at 8.58.09 am

I understand that du only calculates the data in the directory, but it doesn’t take into account data that could be physically in a different directory but from the same user and belongs to the same project.

Can anyone suggest how I can find the extra 120.5 G that could be in other directories?

Thanks in advance,
Chenhui

2

The --group flag will show usage in other project directories:

nci-files-report --group if69 --filesystem gdata --sort-by total_size

but I don’t think that is the problem here, as they wouldn’t be reported as project and group if69.

You could try

find /g/data/if69 --user cj0591 -not -path "/g/data/cj0591/*

to see if there are files you own in another location.

3

Hi Aidan,

Thanks for your prompt reply.

I try find /g/data/if69 -user cj0591 -not -path "/g/data/cj0591/*"

Some data that is under /g/data/if69/.Trash-17286 and belongs to my username cj0591:

scrs_2
scrs_2771×148 17.8 KB

I use du -sh *, which shows:

scrs_1
scrs_1771×148 9.37 KB

The total size is 122 GB, which explains the gap. What are those files? Is Ok to remove this data?

4

I think I’ve seen this before. Do you use ARE? In the past dragging files to the trash/recycle bin on ARE doesn’t actually delete them, but puts them in a hidden folder. This might be similar for you.

You can look into the directory to confirm you don’t need the files anymore, and then delete if appropriate.

1 Like
5

Thanks! I will delete the files that I don’t need.

1 Like
6

Apologies for tagging on the end of this chat. I manage dx2 and there are many reasons I have found for mismatches in the reporting. Is it worthwhile listing these for future players? I am guessing a Project Administration FAQ. I have already shared a few of these ‘tips and tricks’ with colleagues that are taking on the management of other projects. Arguably the potential audience might already be across the issues.

7

If you’d like to start a topic about this I am sure it would be welcome.

8

Yes, I +1 that! Start a thread/topic; it’s a great idea.
A few of us could gather information there! @griff, please tag me if you do!

9

Thanks for this @Aidan! I didn’t know about the .Trash directories!

1 Like