# Rose-cylc stopped working **URL:** \ **Category:** General\ **Tags:** help\ **Created:** [26 August 2026 04:37 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923 "2026-08-26T04:37:54Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [26 August 2026 04:37 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/1 "2026-08-26T04:37:54Z")
## Description of request: Before I went on leave ~3 weeks ago, running AM3 suites was not an issue. ## Environment: NCI ## What executed: Returning to an AM3 development issue, `rose **** ` anything, “rose” is not recognized. persistent-session was restarted - just incase. still nothing module use ~access/modules module load rose “cylc” is not recognized either module load cylc These might not be the best versions, already this is odd as I have never had to load rose/cylc I pulled on dev-n96e (not main) to get the latest configs as I am aware there was stuff with upgrading to 13.8 going on. Unfortunately still falls over, but at least I am back to a python error again. This particular error follows issuing: > rose suite-run: ## Actual results: [FAIL] CYLC\_SUITE\_PROCESS=64437 /g/data/hr22/apps/cylc7/bin/python -s /g/data/hr22/apps/cylc7/cylc\_7.9.7/bin/cylc-run access-am3-configs --host=localhost [FAIL] Try “cylc stop ‘access-am3-configs’” first? gadi:access-am3-configs\> cylc scan gadi:access-am3-configs\> ## Expected results: ## Additional info: ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [26 August 2026 06:39 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/2 "2026-08-26T06:39:57Z")
It looks like you might have conflicting cylc modules. You’re trying to load Rose/Cylc from `/g/data/access/modules`, but the error is reported from the `hr22` cylc. I don’t think the `/g/data/access` distribution has been supported for a while- what happens if you load from `hr22`: ```auto module use /g/data/hr22/modulefiles module load cylc7 ``` ---
**Author:** ![clairecarouge](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/clairecarouge/32/380_2.png) [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)\ **Post date:** [26 August 2026 06:50 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/3 "2026-08-26T06:50:40Z")
---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [26 August 2026 07:48 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/4 "2026-08-26T07:48:27Z")
I purged everything and used everything from hr22. I even tried cylc7-rose, but no dice. There was no ‘access-am3-configs’ cylc job running as reported above, but I thought maybe it is getting stuck, so I removed rm -fr /home/599/jxs599/cylc-run/access-am3-configs/.service/contact It runs with cylc7 and rose from hr22 (well at least rose-cylc does, and it starts sub-ing model jobs). Considering this might have been the problem all along, I did it all over again with no modules manually loaded, picking those up from the suite I guess. rose/cylc gets way further than before, but ultimately crashes before it qsubs model components. I think it cant find ports. Anyway, I can use hr22 modules for now. ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [26 August 2026 08:08 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/5 "2026-08-26T08:08:32Z")
To clarify- do you mean it is working with `hr22` modules, or still failing even when using `hr22`? And you definitely restarted the persistent session with the same persistent session name? I know I’ve had many issues if I ever tried to change the name of my persistent session, things around my `.ssh/known_hosts`. ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [26 August 2026 10:52 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/6 "2026-08-26T10:52:52Z")
I meant it is working with hr22. Well I’ve run into other problems, so it doesn’t exactly work , but it isnt this rose-cylc problem. Thanks Lachlan. ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 02:14 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/7 "2026-08-27T02:14:13Z")
This is a beaut - using the hr22 module cylc7-rose, the suite launched the job. late yesterday I got to the FCM build starting but compilation dropping out because of multiple sources for UKCA stuff etc. I noticed in the NEW config that you have put libraries (or at least distinct source paths, executables) in place for UKCA etc. Expecting that you must’ve removed the from the UM src/ I pulled from AM3-dev to pick them up. Actually I had to re-clone as it screwed it self up. Anyway, it all “seemed” to build, at least suite GUI registers “succeeded” and went on to trigger recon (where it hangs with retrying submission). There was a glitch on gadi this morning where qstat couldn’t find the queue etc AND the sporadically re-curring issue where I have to HOLD everything after building - I didnt do this - so I decided just to run it again. > [INFO] export CYLC\_VERSION=7.9.6 > [INFO] export ROSE\_ORIG\_HOST=gadi-login-01.gadi.nci.org.au > [INFO] export ROSE\_SITE=nci > [INFO] export ROSE\_VERSION=2019.01.7 > [INFO] create: log.20260827T015208Z > [INFO] delete: log > [INFO] symlink: log.20260827T015208Z \<= log > [INFO] log.20260826T231729Z.tar.gz \<= log.20260826T231729Z > [FAIL] [Errno 16] Device or resource busy: ‘log.20260826T231729Z/suite/.nfs8091450e015a3f90000091e7’ Odd but I have had .nfs files get stuck before after a glitch. SO I went to where the logs usually are to try to forcefully remove it. > gadi:cylc-run\> pwd -P > /scratch/p66/jxs599/cylc-run > gadi:cylc-run\> ls -ltr | tail -1 > drwxr-s— 11 jxs599 p66 4096 Jul 4 12:19 access-am3-configs-Failed-recon/ My comment above “seemed” to build - success reported by the GUI, however the directory, > access-am3-configs/ was never made. So where has it put everything? I looked in: > gadi:tmp\> pwd -P > /scratch/p66/jxs599/tmp > gadi:tmp\> ls -ltr | tail -7 > drwx–S— 2 jxs599 p66 4096 Aug 26 18:11 cylc-9bOf0q/ > drwx–S— 2 jxs599 p66 4096 Aug 27 08:45 ssh-lh55FX0sFd0M/ > drwx–S— 2 jxs599 p66 4096 Aug 27 08:45 ssh-oxWcPcmlMLQj/ > drwx–S— 2 jxs599 p66 4096 Aug 27 08:46 ssh-GUewb1684xVJ/ > drwx–S— 2 jxs599 p66 4096 Aug 27 08:46 ssh-VzYAPcxyDCn0/ > drwx–S— 2 jxs599 p66 4096 Aug 27 09:17 cylc-yIXOI2/ > drwx–S— 2 jxs599 p66 4096 Aug 27 09:20 ssh-VrECuqu7bdcC/ > gadi:tmp\> but all the dirs created today are basically empty Any ideas? @lachlanswhyborn @MartinDix ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [27 August 2026 02:34 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/8 "2026-08-27T02:34:48Z")
Can you go over what happens if you go from the start? So start by loading cylc via `hr22`, cloning the config to a specific dir, and then `rose suite-run` from that dir? So we can follow precisely what steps you’re doing to replicate the issue. Can you also point to the installation directory for the current suite run? ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 02:50 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/9 "2026-08-27T02:50:45Z")
ok - so which branch should I use > remotes/origin/dev-n96e > remotes/origin/dev-n96e-vn13.8 > remotes/origin/release-n96e Eventually I would prefer the FCM build that picks up local code. In the first instance I’ll just try a more std config and just edit for p66 operation ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [27 August 2026 02:52 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/10 "2026-08-27T02:52:09Z")
May as well start with `dev-n96e`. See if it runs in the normal way (with a deployed executable), and then if that’s successful, change to the in-suite build. ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 03:08 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/11 "2026-08-27T03:08:17Z")
perhaps to be expected; it doesn’t change that the FS is caught up > gadi:roses\> pwd -P > > /home/599/jxs599/roses > gadi:roses\> git clone [git@github.com](mailto:git@github.com):ACCESS-NRI/access-am3-configs.git > > Cloning into ‘access-am3-configs’… > > … > > …. > > Resolving deltas: 100% (2023/2023), done. > gadi:roses\> cd access-am3-configs > gadi:access-am3-configs\> git switch dev-n96e > branch ‘dev-n96e’ set up to track ‘origin/dev-n96e’. > Switched to a new branch ‘dev-n96e’ > gadi:access-am3-configs\> module load cylc7-rose > > Loading cylc7-rose/7.9.6\_2019.01.7 > > Loading requirement: python2/2.7.17 > gadi:access-am3-configs\> rose suite-run > [INFO] export CYLC\_VERSION=7.9.6 > [INFO] export ROSE\_ORIG\_HOST=gadi-login-01.gadi.nci.org.au > [INFO] export ROSE\_SITE=nci > [INFO] export ROSE\_VERSION=2019.01.7 > [INFO] create: log.20260827T025650Z > [INFO] delete: log > [INFO] symlink: log.20260827T025650Z \<= log > [INFO] log.20260827T021330Z.tar.gz \<= log.20260827T021330Z > [INFO] delete: log.20260827T021330Z/ > [INFO] log.20260826T231729Z.tar.gz \<= log.20260826T231729Z > [FAIL] [Errno 16] Device or resource busy: ‘log.20260826T231729Z/suite/.nfs8091450e015a3f90000091e7’ wait … ah ok there is a distinct cylc-run in home AND scratch. I’ll remove this access-am3-configs dir and VOILA - it is launching again. Assuming this completes - I’ll retry the FCM build and report back ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 03:50 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/12 "2026-08-27T03:50:25Z")
DOH! shouldn’t have assumed - recon fails. email report shows: > PBS Job Id: 177583236.gadi-pbs > Job Name: recon.19820101T > Post job file processing error; job 177583236.gadi-pbs on host gadi-cpu-clx-1973 There isnt a lot to work with here: > gadi:19820101T0000Z\> ls -altr /home/599/jxs599/cylc-run/access-am3-configs/log/job/19820101T0000Z/recon/01/ > total 20 > drwxr-xr-x 3 jxs599 p66 4096 Aug 27 13:21 ../ > -rwxr-xr-x 1 jxs599 p66 3798 Aug 27 13:21 job\* > -rw-r–r-- 1 jxs599 p66 158 Aug 27 13:21 job-activity.log > -rw-r–r-- 1 jxs599 p66 242 Aug 27 13:23 job.status > -rw-r–r-- 1 jxs599 p66 0 Aug 27 13:23 job.out .out is empty and there is no .err file. Preceding initial\_\* tasks seem to have worked. > /home/599/jxs599/cylc-run/access-am3-configs/share/data/ shows both .astarts are there /home/599/jxs599/cylc-run/access-am3-configs/work/19820101T0000Z/recon/pe\_output/am3.fort6.pe0 looks pretty normal - nothing stands out to me anyway I am curious though what it is executing seeing as though nothing was built for the neon OR model like: > From a month ago: > /scratch/p66/jxs599/cylc-run/access-am3-configs-Failed-recon/share/fcm\_make\_um/build-recon/bin/um-recon.exe ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [27 August 2026 10:02 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/13 "2026-08-27T10:02:27Z")
Which project are installing the suite in? I’ll see if I can take a look at the logs ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 10:24 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/14 "2026-08-27T10:24:12Z")
p66 I tried turning off the reconf as there is a .astart file present which I think the model should be fine. The model seems to start. it creates all the pe\_output files. although \*.000 is empty. individual processor files look like they have started, but stop short of time-stepping. Very weird. Then it just stops. It doesn’t leave a .err file. The .out file is empty. ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [27 August 2026 11:53 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/15 "2026-08-27T11:53:45Z")
I don’t see anything modified recently under your `p66` `cylc-run` directory- latest was July? ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 August 2026 22:47 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/16 "2026-08-27T22:47:30Z")
This might even be the problem. It has been this way for years though. At least the heavier parts. I suppose it was set up that way because $HOME/ wasn’t big enough to contain cylc-run. Especially, > /home/599/jxs599/cylc-run/access-am3-configs/share/data/History\_Data where output goes. You can easily shut down the queue. It seems that they have increased the size of $HOME directories. Back in the day we only had few Gb each. Even still, runs generating Tb of data cant sit here. but even /home/599/jxs599/cylc-run/access-am3-configs/share/fcm\_make\_um/ is/was problematic. Not required for a SPACK build though ================================================================= this is the version of cylc-run/ in $HOME > gadi:cylc-run\> pwd -P > /home/599/jxs599/cylc-run > gadi:cylc-run\> ls -altr | tail -3 > … > … > drwxr-x— 11 jxs599 p66 4096 Aug 27 19:12 access-am3-configs/ this is the version of cylc-run/ in $SCRATCH; where the $HOME version is linked to (at least in part). Note: There is no new instance of access-am3-configs/. Sorry, I never really acknowledged where (or how) the demarkation line was. I knew it was doing it, and it was just linked, but it just became muscle memory to know which version to go to. > gadi:cylc-run\> pwd -P > /scratch/p66/jxs599/cylc-run > gadi:cylc-run\> ls -altr | tail -3 > … > … > drwxr-s— 11 jxs599 p66 4096 Aug 28 01:05 access-am3-configs\_mergedToday?/ ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [27 August 2026 23:32 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/17 "2026-08-27T23:32:33Z")
Ahh I suspect I may know what is happening. Is this a configuration that was installed some time ago, and you’re reusing it for your test runs? And you’ve pulled the most recent version of `dev-n96e`? With CABLE changing to be included as a library, there are additional tasks in the graph to retrieve and build CABLE source. The `build_cable` task is a dependency of the `fcm_make_um` task, but perhaps your task graph was not rebuilt, so it’s in a weird state where `fcm_make_um` thinks it needs `build_cable` to run, but `build_cable` is not in your task graph. Apparently, Cylc7 graphs don’t update when you `reload`, but Cylc8 ones do. ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [28 August 2026 01:51 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/18 "2026-08-28T01:51:20Z")
I was worried this might happen so I cloned it fresh yesterday: > gadi:access-am3-configs\> git reflog show --date=iso > 1ea95f5 (HEAD → dev-n96e, origin/dev-n96e) HEAD@{2026-08-27 12:54:26 +1000}: checkout: moving from dev-n96e to dev-n96e > 1ea95f5 (HEAD → dev-n96e, origin/dev-n96e) HEAD@{2026-08-27 12:54:00 +1000}: checkout: moving from main to dev-n96e > 2ec0c6f (origin/release-n512e-aeroclim, origin/main, origin/HEAD, main) HEAD@{2026-08-27 12:38:14 +1000}: clone: from [github.com](http://github.com):ACCESS-NRI/access-am3-configs.git > gadi:access-am3-configs\> git branch > > - dev-n96e > main > gadi:access-am3-configs\> git diff > diff --git a/rose-suite.conf\_nci\_gadi b/rose-suite.conf\_nci\_gadi > index e2bee98..81fdba7 100644 > — a/rose-suite.conf\_nci\_gadi > +++ b/rose-suite.conf\_nci\_gadi > @@ -1,6 +1,6 @@ > [jinja2:suite.rc] > -STORAGE\_PROJECT=‘${PROJECT}’ > -COMPUTE\_PROJECT=‘${PROJECT}’ > +STORAGE\_PROJECT=‘p66’ > +COMPUTE\_PROJECT=‘p66’ > BITCOMP\_NRUN=false > DATAM=‘History\_Data’ > EXPT\_AEROSOLS=‘aeroprog’ > gadi:access-am3-configs\> module use /g/data/hr22/modulefiles > gadi:access-am3-configs\> module load cylc7-rose > gadi:access-am3-configs\> rose suite-run The graph shows: ![Screenshot 2026-08-28 at 11.47.10 AM](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/b/bb04861e204c68798e6f4ed0b8132d9280963363.png) At some stage yesterday I was using an updated config. Not entirely sure, but I think this might’ve been the dev-n96e-vn13.8 branch and it did show a graph that looked more in line with sources coming from libraries etc. ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [28 August 2026 04:00 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/19 "2026-08-28T04:00:19Z")
That’s the expected graph for the default configuration, that doesn’t do the in-suite build. Do the `install_` tasks ever finish? If there are issues with the directory Cylc thinks it should install to and the mounted directories, this can happen, but I don’t see any issues with that in your setup. ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [28 August 2026 04:14 UTC](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923/20 "2026-08-28T04:14:35Z")
yeah they always do. This time around the recon finished as well. Atmos ran until dt=525. But there are no (or sometimes empty) log files. I haven’t looked at all the pe\_output yet to see where it failed [Next page](https://forum.access-hive.org.au/t/rose-cylc-stopped-working/6923.md?page=2)