# Trouble shooting a stalled ACCESS-CM2 suite **URL:** \ **Category:** Earth System\ **Tags:** help\ **Created:** [11 December 2024 06:59 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988 "2024-12-11T06:59:20Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [11 December 2024 06:59 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/1 "2024-12-11T06:59:20Z")
I have had a few CM2 suites hang/not succeed today and I can’t figure out why. This the GUI interface for one of my simulations that doesn’t want to run (u-dl669). ![Screenshot 2024-12-11 at 5.41.11 PM](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/5/5b02cd3af733062540b965e364475879428d84ba.png) Nothing is showing up as red/failed in the GUI but I also don’t have any active jobs running for this suite and if I look at some of my output logs, then several tasks do show up as failed. For example, this is the `job.err` file for the `fcm_make_um` task. ![Screenshot 2024-12-11 at 5.44.20 PM](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/6/61376fbdf528c296e7418bf2f036d2eab300ce07.png) I’m not sure if it’s related to this [recent post](https://forum.access-hive.org.au/t/access-cm2-suite-run-error/3938/14). I’ve tried stopping the suite, restarting the suite, reloading the suite, deleting my local copy and rechecking out the suite, and I get the same result. I definitely don’t have the same folders in my ~/cylc-run/u-dl669/share/ directory that my successful runs have, but I’m unsure why. Does anyone have any suggestions? I would be happy to start over (since nothing has actually run) and I have tried the HARD restart option, but that also didn’t work for me. ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [11 December 2024 22:17 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/2 "2024-12-11T22:17:30Z")
- Check if `~/cylc-run/u-dl669` exists - Check if `~/cylc-run/u-dl669/share` is a symlink and where it points (e.g. with `readlink -f`) - Check the ‘job’ script has the correct storage flags (hr22 and wherever share points to) ---
**Author:** ![wghuneke](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/wghuneke/32/102_2.png) [@wghuneke](https://forum.access-hive.org.au/u/wghuneke)\ **Post date:** [11 December 2024 22:18 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/3 "2024-12-11T22:18:25Z")
Have you tried a `rose suite-clean`? This has worked for me in the past. ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [11 December 2024 22:47 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/4 "2024-12-11T22:47:46Z")
Yes, `~/cylc-run/u-dl669` exists. The `~/cylc-run/u-dl669/share` is not a symlink and the job script does have the hr22 flag. ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [11 December 2024 22:52 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/5 "2024-12-11T22:52:49Z")
I’ve just run the `rose suite-clean`. Should I now just try rerun again with `rose suite-run`? ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [11 December 2024 23:10 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/6 "2024-12-11T23:10:39Z")
I tried this (`rose suite-clean` and then `rose suite-run`) and the exact same thing has happened again. No errors are thrown in the GUI, the jobs show up as green and ‘submitted’ but there’s no jobs running on gadi. ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [11 December 2024 23:47 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/7 "2024-12-11T23:47:36Z")
Hi Hannah. Have you checked you disk quota in your home directory and any relevant projects? I’ve had issues with rose/cylc in the past that have often been caused by disk quota issues. ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [11 December 2024 23:56 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/8 "2024-12-11T23:56:30Z")
@Paul.Gregory my disk quota was exceeded yesterday and so I had wondered if this could be a problem. But then I removed data and copied a new suite (once my disk quota was under the limit) and the exact same thing happened, so I had ruled it out on this basis… Maybe I shouldn’t have? If that’s the problem, are these particular suites doomed to fail or is there a way to recover them so to speak? ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [12 December 2024 00:12 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/9 "2024-12-12T00:12:25Z")
Assuming qstat shows the job in the queue, qstat -f will show you more info. Under “comments” it will tell you why your job is being held. It could even be a group quota issue ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [12 December 2024 00:16 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/10 "2024-12-12T00:16:32Z")
> [@Jhan](#): > > qstat -f Thanks @Jhan. The issue was that I didn’t even have any jobs submitted/in queue even though the GUI was telling me that I did. I’m trying to rerun now (using `rose suite-run --name dl669_try`) and can see that some jobs are being held because `Not enough free nodes available`. I’ll just wait and see if this reattempt works when the jobs make it through the queue. Thanks! ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [12 December 2024 04:51 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/11 "2024-12-12T04:51:35Z")
I don’t think the home disk quota was the problem. I think the problem is related to the project I was trying to run the simulations on. All my runs that have successfully got past the build stage have been run on `e14`. All my runs that are ‘held’ including a new suite that I created this afternoon have been run on `ol01`. If I change the project from `ol01` to `e14`, the the simulation will at least run the build process. Is there any particular reason for this? `ol01` still has compute time and doesn’t seem to be over the disk quota on scratch or gdata either. ---
**Author:** ![clairecarouge](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/clairecarouge/32/380_2.png) [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)\ **Post date:** [12 December 2024 17:47 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/12 "2024-12-12T17:47:04Z")
Storage flag somewhere missing ol01 maybe? ---
**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\ **Post date:** [16 December 2024 04:27 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/13 "2024-12-16T04:27:16Z")
I’ve tried adding storage flags to `gdata/ol01` and `scratch/ol01` and two particular suites (`u-dl669` and `u-dl670`) always seems to stall. However, I’ve been able to run a newer suite on `ol01` which seems to be working. I guess the `u-dl669` suite was unhappy as it was copied when my home disk quota was exceeded. I don’t really understand why `u-dl670` also has this problem as I copied that suite once my disk quota was cleaned up and under limit. I also don’t know why deleting and reinstalling the suites doesn’t work now that my disk quota is under limit, but in any case I can run newer projects on `ol01` if I explicitly include the storage flags in the `suite.rc` file so I guess this can be closed off? Learnings: Don’t checkout or copy new suites if your disk quota is close to limit. ---
**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\ **Post date:** [14 January 2025 21:51 UTC](https://forum.access-hive.org.au/t/trouble-shooting-a-stalled-access-cm2-suite/3988/14 "2025-01-14T21:51:37Z")
Closed on 14 January 2025 21:51 UTC