# OAS and RAS suites stalling for ~1 day on Gadi and then failing **URL:** \ **Category:** rAM3/Regional Nesting Suite\ **Tags:** help, access-ram3\ **Created:** [4 May 2026 01:18 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364 "2026-05-04T01:18:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 01:18 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/1 "2026-05-04T01:18:42Z")
Hi all, I’m running test-case simulations for the Lismore flood setup using ACCESS-rAM3, and I’m encountering an issue. The OAS and RAS suites appear to start normally, but the jobs seem to sit for an extended period (up to ~1 day) without progressing, and then eventually fail or get terminated. In some cases, Cylc reports a submission timeout (P1D), and the suite shuts down automatically. What I’ve checked so far: - Persistent session is active and accessible (SSH into the session host works) - Cylc suite starts and registers correctly - No obvious errors in initial suite startup logs - Jobs sometimes appear to be submitted but do not transition to running Has anyone experienced similar behaviour with OAS/RAS suites? Any suggestions on what to check (queue settings, resource requests, or Cylc configuration) would be really helpful. Thanks in advance! Moulik ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 01:22 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/2 "2026-05-04T01:22:12Z")
OAS ![Screenshot 2026-05-04 110337](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/1/12d38021aa9a971a76d800255c461079b439cf57.png) RAS ![Screenshot 2026-05-04 110450](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/b/b0b6bf3ff1e5cda9772aff1a8ea66f45eca97b22.png) FYR: exactly look like this. ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [4 May 2026 02:50 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/3 "2026-05-04T02:50:45Z")
`qstat -f` will show a comment field that may explain the reason a job is not running - could be something like your project is over quota. Also there is system maintenance tomorrow, if a job has a long enough walltime to conflict with the maintenance period it won’t be started until after the maintenance. ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 03:19 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/4 "2026-05-04T03:19:02Z")
Hi @Scott, Thanks for your reply. I did check the status of the tasks submitted in both the suites - it’s indicated that the jobs are finished. See attached. [mm6452@gadi-cpu-bdw-0015 ~]$ qstat -f 167665179.gadi-pbs qstat: 167665179.gadi-pbs Job has finished, use -x or -H to obtain historical job information [1]+ Done rose suite-gcontrol --name=u-dk517 [mm6452@gadi-cpu-bdw-0015 ~]$ rose suite-gcontrol --name=u-bu503 & [1] 131174 [mm6452@gadi-cpu-bdw-0015 ~]$ qstat -f 167665043.gadi-pbs qstat: 167665043.gadi-pbs Job has finished, use -x or -H to obtain historical job information [1]+ Done rose suite-gcontrol --name=u-bu503 But I am wondering it’s still showing submitted in both the GUI. Any reasons? Also plenty of tasks are on waiting. ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [4 May 2026 03:21 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/5 "2026-05-04T03:21:34Z")
Hi @Scott, just a bit more background info here too. Moulik had this issue for jobs submitted after the gadi glitch last week. We wondered if it was something to do with the system failure (there were reports of long queue times) so killed the suites and resubmitted them on Friday. It could be that they won’t now start at this point due to the maintenance shutdown tomorrow, but I don’t think that’s the root cause of what’s going on. ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [4 May 2026 03:23 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/6 "2026-05-04T03:23:34Z")
@Moulik which project are you running on? ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 03:23 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/7 "2026-05-04T03:23:58Z")
Thanks @bethanwhite for mentioning it. I am running it on if69. ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [4 May 2026 03:26 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/8 "2026-05-04T03:26:44Z")
I don’t have permission to check the status of that project. Let me see if I can find someone who has if69 permissions to check that there are still resources and disk space on it. ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 03:28 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/9 "2026-05-04T03:28:46Z")
Okay that would be great! thanks! ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [4 May 2026 03:42 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/10 "2026-05-04T03:42:09Z")
If you right click on the tasks in the GUI there should be a ‘poll’ option that will check the job’s current status in PBS and update them in the GUI - otherwise the GUI only gets updated when the job actually starts. To see your project’s current status try ```auto nci_account -P if69 ``` ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [4 May 2026 03:54 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/11 "2026-05-04T03:54:25Z")
Pao has reported on if69 for us - all looks fine. ```auto [pc2687@gadi-login-06 ~]$ nci_account -P if69 Usage Report: Project=if69 Period=2026.q2 ============================================================= Grant: 1.25 MSU Used: 197.59 KSU Reserved: 4.38 KSU Avail: 1.05 MSU Storage Usage Report: Project=if69 ============================================================= Filesystem Used iUsed Allocation iAllocation gdata6 14.13 TiB 456.54 K 20.00 TiB 1.77 M scratch1 14.76 TiB 1.50 M 20.00 TiB 1.77 M ============================================================= ``` ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [4 May 2026 04:14 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/12 "2026-05-04T04:14:16Z")
OK, yeah that’s fine. Suggest you retry the workflow and keep an eye on `qstat -f` output if it’s taking a long time to start. ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 04:19 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/13 "2026-05-04T04:19:16Z")
By using the poll option on my current job on OAS suite I found that - job (01) submission failed (polled). and then I checked the job error file: Using the cylc session mlftcr.mm6452.if69.ps.gadi.nci.org.au Loading cylc7/24.03 Loading requirement: mosrs-setup/2.0.1 [FAIL] [Errno 2] No such file or directory: ‘/home/581/mm6452/cylc-run/u-dk517/share’ mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-dk517’: File exists mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-dk517’: File exists mkdir: cannot create directory ‘/home/581/mm6452/cylc-run/u-dk517’: File exists 2026-05-03T13:59:08Z CRITICAL - failed/ERR ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [4 May 2026 04:26 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/14 "2026-05-04T04:26:34Z")
Hi @Moulik, Have you checked out and run u-dk517 to make your own OSTIA ancillary files? If not, if you don’t want OSTIA can you please follow [these instructions](https://access-ram3-configs.access-hive.org.au/initial_conditions/replace_sst_seaice/) and set USE\_OSTIA to FALSE … ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [4 May 2026 04:28 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/15 "2026-05-04T04:28:46Z")
Sorry just re-read your post. It seems like for some reason your jobs are trying to write to your home directory, they need to be writing to scratch. I haven’t got Gadi open at the moment, @Paul.Gregory or @bethanwhite can you please tell @Moulik how to fix his output directory (if that is it) … ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 04:33 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/16 "2026-05-04T04:33:25Z")
Hi @cbengel - thanks for the replies. Please take your time - we can have a look when you are back at Gadi. Otherwise I am in contact with @bethanwhite and hope to get some help soon. No worries! ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [4 May 2026 04:57 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/17 "2026-05-04T04:57:57Z")
Hi @cbengel does a successful OAS run looks like that? mm6452@gadi-cpu-bdw-0015 u-dk517]$ cd .. [mm6452@gadi-cpu-bdw-0015 roses]$ cd .. [mm6452@gadi-cpu-bdw-0015 ~]$ cylc cat-log u-dk517 2026-05-04T04:49:05Z INFO - Suite server: url=[https://gadi-cpu-bdw-0015.gadi.nci.org.au:43046/](https://gadi-cpu-bdw-0015.gadi.nci.org.au:43046/) pid=180337 2026-05-04T04:49:05Z INFO - Run: (re)start=0 log=1 2026-05-04T04:49:05Z INFO - Cylc version: 7.9.9 2026-05-04T04:49:05Z INFO - Run mode: live 2026-05-04T04:49:05Z INFO - Initial point: 1 2026-05-04T04:49:05Z INFO - Final point: 1 2026-05-04T04:49:05Z INFO - Cold Start 1 2026-05-04T04:49:07Z INFO - [ostia\_netcdf\_to\_pp.1] -submit-num=01, owner@host=gadi.nci.org.au 2026-05-04T04:49:08Z INFO - [client-command] get\_latest\_state mm6452@gadi-cpu-bdw-0015.gadi.nci.org.au:cylc-gui 0906f91f-d888-4b97-8f0a-d62fa5e958e1 2026-05-04T04:49:09Z INFO - [ostia\_netcdf\_to\_pp.1] status=ready: (internal)submitted at 2026-05-04T04:49:09Z for job(01) 2026-05-04T04:49:09Z INFO - [ostia\_netcdf\_to\_pp.1] -health check settings: submission timeout=P1D 2026-05-04T04:49:39Z INFO - Command succeeded: poll\_tasks([u’ostia\_netcdf\_to\_pp.1’], poll\_succ=False) 2026-05-04T04:49:39Z INFO - Processing 1 queued command(s) - poll\_tasks([u’ostia\_netcdf\_to\_pp.1’], poll\_succ=False) 2026-05-04T04:49:41Z INFO - [ostia\_netcdf\_to\_pp.1] status=submitted: (polled)submitted at 2026-05-04T04:49:09Z for job(01) 2026-05-04T04:49:56Z INFO - [ostia\_netcdf\_to\_pp.1] status=submitted: (received)started at 2026-05-04T04:49:55Z for job(01) 2026-05-04T04:49:56Z INFO - [ostia\_netcdf\_to\_pp.1] -health check settings: execution timeout=PT11M, polling intervals=2\*PT2M,PT7M,… 2026-05-04T04:50:58Z INFO - [ostia\_netcdf\_to\_pp.1] status=running: (received)succeeded at 2026-05-04T04:50:57Z for job(01) 2026-05-04T04:51:03Z INFO - [ancil\_sst\_seaice.1] -submit-num=01, owner@host=gadi.nci.org.au 2026-05-04T04:51:06Z INFO - [ancil\_sst\_seaice.1] status=ready: (internal)submitted at 2026-05-04T04:51:06Z for job(01) 2026-05-04T04:51:06Z INFO - [ancil\_sst\_seaice.1] -health check settings: submission timeout=P1D 2026-05-04T04:51:50Z INFO - Command succeeded: poll\_tasks([u’ancil\_sst\_seaice.1’], poll\_succ=False) 2026-05-04T04:51:50Z INFO - Processing 1 queued command(s) - poll\_tasks([u’ancil\_sst\_seaice.1’], poll\_succ=False) 2026-05-04T04:51:52Z INFO - [ancil\_sst\_seaice.1] status=submitted: (polled)submitted at 2026-05-04T04:51:06Z for job(01) 2026-05-04T04:52:04Z INFO - [ancil\_sst\_seaice.1] status=submitted: (received)started at 2026-05-04T04:52:02Z for job(01) 2026-05-04T04:52:04Z INFO - [ancil\_sst\_seaice.1] -health check settings: execution timeout=PT20M, polling intervals=PT11M,PT2M,PT7M,… 2026-05-04T04:53:30Z INFO - [ancil\_sst\_seaice.1] status=running: (received)succeeded at 2026-05-04T04:53:30Z for job(01) 2026-05-04T04:53:32Z INFO - Command succeeded: poll\_tasks([u’ancil\_sst\_seaice.1’], poll\_succ=False) 2026-05-04T04:53:32Z INFO - Processing 1 queued command(s) - poll\_tasks([u’ancil\_sst\_seaice.1’], poll\_succ=False) 2026-05-04T04:53:42Z INFO - Suite shutting down - AUTOMATIC 2026-05-04T04:53:48Z INFO - DONE ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [4 May 2026 05:00 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/18 "2026-05-04T05:00:33Z")
It looks promising - Gadi session still hasn’t started so I can’t compare and I don’t usually look at the polling. If you go to the `OSTIA_OUTPUT` directory if it succeeded you will find the ancillary files there. ---
**Author:** ![Moulik](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/moulik/32/4308_2.png) [@Moulik](https://forum.access-hive.org.au/u/Moulik)\ **Post date:** [6 May 2026 00:22 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/19 "2026-05-06T00:22:21Z")
Hi @cbengel I did check the OSTIA\_OUTPUT directory and I can see the ancillary files of 25th, 26th, and 27th. So I guess I ran the OAS successfully. When I started running the RAS suite this morning, I encountered this error on my ANCIL run: ERROR: file not found: /home/581/mm6452/cylc-run/u-bu503/log/job/1/install\_ugants/01/job.err Do any one of you have any idea where I am doing the wrong? Thanks! ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [6 May 2026 00:24 UTC](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364/20 "2026-05-06T00:24:14Z")
@Moulik For any job starting today my first suspicion would be problems due to Gadi going down. Please check that you have restarted your persistent session by running `persistent-sessions list`. If it isn’t there you need to restart it using the same name you used last time. Do that first then we can move on to other suggestions … [Next page](https://forum.access-hive.org.au/t/oas-and-ras-suites-stalling-for-1-day-on-gadi-and-then-failing/6364.md?page=2)