Hi @Moulik are you up and running now?
Hey all,
I finally managed to get the RAS suite running (and so does RNS on the weekend!), so I wanted to share the issue and solution here in case it helps others.
The main problem was that the RAS suite jobs were repeatedly failing during tasks such as install_ugants, with errors like:
[FAIL] [Errno 2] No such file or directory:
'/home/581/mm6452/cylc-run/u-bu503/share'
and with:
mkdir: cannot create directory ... File exists
I had no idea, but the actual issue turned out to be related to PBS storage flags as mentioned by @Scott at some point, thanks!!!
After inspecting the generated PBS job script:
cat ~/cylc-run/u-bu503/log/job/1/install_ugants/01/job
I noticed that the storage line did not include my actual project scratch space (scratch/if69). The generated job contained something like:
#PBS -l storage=gdata/access+gdata/hr22+...
but was missing:
scratch/if69
As a result, compute-node jobs could not properly access the suite run directories under $HOME/cylc-run/..., which caused the missing share/errors.
The fix was to edit:
site/nci-gadi/suite-adds.rc
and ensure the PBS storage flags explicitly included the correct project storage:
scratch/if69
I also found that some storage settings were using:
...{{environ.get('PROJECT','NO_PROJECT')}}
which was not resolving correctly in my case.
After correcting the storage flags, the RAS suite ran successfully. Thanks very much!
Hi @Moulik,
In u-bu503, the scratch/$PROJECT is included under “suite conf” → “Regional Ancillary Suite” → “NCI Site Options” → NCI_STORAGE = scratch/$PROJECT.
This is why I was asking if if69 was the project you were running the job under.
Did that option perhaps get deleted?
Anyway, thank you for summarising your experience. Hopefully others will find it of interest.
Hi @cbengel , thank you for checking in.
I have a quick question for you (if you can remember). I participated in the ACCESS-rAM3 hands-on at AMOS 2026, and successfully ran the test case. But I can’t remember that - did we use ERA5 SST or OSTIA SST for the test case run there?
Because this time I used the OSTIA SST and I have a bit different rainfall patterns, so I am a bit curious. Thanks in advance!
The option was still there - but (may be) somehow I did something wrong, so it was not flagged into my PBS storage. But finally I found it by checking the job-activity.log file.
Cool.
We did use pre-computed and stored OSTIA files in the tutorial.
The discussion was a bit confusing because I think something has gone wrong with the working scratch/$PROJECT setting - I do think something must have been deleted or changed.
Either way, hopefully you gained some knowledge from the hunting down of the setting. I will ask the user support team to close this ticket now.
Thanks @cbengel.
Before closing I am just going to summarise at least the latest issue as that could be something other users encouter.
When running a cylc suite, if you encounter failures of the type:
Using the cylc session mlftcr.mm6452.if69.ps.gadi.nci.org.au
Loading cylc7/24.03
Loading requirement: mosrs-setup/2.0.1
[FAIL] [Errno 2] No such file or directory: ‘/home/581/mm6452/cylc-run/u-bu503/share’
mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-bu503’: File exists
mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-bu503’: File exists
mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-bu503’: File exists
2026-05-06T02:00:09Z CRITICAL - failed/ERR
These indicate a storage flag issue. ACCESS-rAM3 is only designed to run from your default project on Gadi. This project is listed under ~/.config/gadi-login.conf.
To diagnose the issue further, follow @Scott’s tip:
Check where your cylc run is putting its files with
readlink -f /home/581/mm6452/cylc-run/u-bu503/shareMake sure whichever disk this points to (e.g.
/scratch/ab12) is in the PBS storage flags (you can set the storage flags inrose-suite.confusing e.g.NCI_STORAGE=scratch/ab12+gdata/de34or add them insite/nci-gadi/flow-adds.cylc)
Then you can follow @Moulik 's step-by-step description to try and fix the issue.
A further update on this issue, as I’m sitting with Moulik now and we ran into exactly the same problem again.
The issue is that u-bu503 (ancillary suite) tasks do not always respect the NCI_PROJECT=“” in rose-suite.conf, but use the ~/.config/gadi-login.conf default project instead. Moulik ran across this issue with install_ugants (which is a newer task developed by the Met Office for LFRiC and has not yet been fully ported to NCI use). His default project had no compute, so the submission of the task failed (but without clear message why).
There isn’t much guidance at all for users on ~/.config/gadi-login.conf, and it is quite important for 21st Century Weather users to have the flexibility to run on different projects (as there are 4 compute projects within the centre, and others for bespoke projects).
So it would be useful to ensure the next release of ACCESS-rAM3 tasks do respect the explicit project defined in NCI_PROJECT=“” in rose-suite.conf.