It looks like you might have conflicting cylc modules. You’re trying to load Rose/Cylc from /g/data/access/modules, but the error is reported from the hr22 cylc. I don’t think the /g/data/access distribution has been supported for a while- what happens if you load from hr22:
module use /g/data/hr22/modulefiles
module load cylc7
It runs with cylc7 and rose from hr22 (well at least rose-cylc does, and it starts sub-ing model jobs).
Considering this might have been the problem all along, I did it all over again with no modules manually loaded, picking those up from the suite I guess. rose/cylc gets way further than before, but ultimately crashes before it qsubs model components. I think it cant find ports. Anyway, I can use hr22 modules for now.
To clarify- do you mean it is working with hr22 modules, or still failing even when using hr22? And you definitely restarted the persistent session with the same persistent session name? I know I’ve had many issues if I ever tried to change the name of my persistent session, things around my .ssh/known_hosts.
This is a beaut - using the hr22 module cylc7-rose, the suite launched the job. late yesterday I got to the FCM build starting but compilation dropping out because of multiple sources for UKCA stuff etc. I noticed in the NEW config that you have put libraries (or at least distinct source paths, executables) in place for UKCA etc. Expecting that you must’ve removed the from the UM src/ I pulled from AM3-dev to pick them up. Actually I had to re-clone as it screwed it self up. Anyway, it all “seemed” to build, at least suite GUI registers “succeeded” and went on to trigger recon (where it hangs with retrying submission). There was a glitch on gadi this morning where qstat couldn’t find the queue etc AND the sporadically re-curring issue where I have to HOLD everything after building - I didnt do this - so I decided just to run it again.
Can you go over what happens if you go from the start? So start by loading cylc via hr22, cloning the config to a specific dir, and then rose suite-run from that dir? So we can follow precisely what steps you’re doing to replicate the issue.
Can you also point to the installation directory for the current suite run?
May as well start with dev-n96e. See if it runs in the normal way (with a deployed executable), and then if that’s successful, change to the in-suite build.
I tried turning off the reconf as there is a .astart file present which I think the model should be fine. The model seems to start. it creates all the pe_output files. although *.000 is empty. individual processor files look like they have started, but stop short of time-stepping. Very weird. Then it just stops. It doesn’t leave a .err file. The .out file is empty.
This might even be the problem. It has been this way for years though. At least the heavier parts. I suppose it was set up that way because $HOME/ wasn’t big enough to contain cylc-run. Especially,
where output goes. You can easily shut down the queue. It seems that they have increased the size of $HOME directories. Back in the day we only had few Gb each. Even still, runs generating Tb of data cant sit here.
gadi:cylc-run> ls -altr | tail -3
…
…
drwxr-x— 11 jxs599 p66 4096 Aug 27 19:12 access-am3-configs/
this is the version of cylc-run/ in $SCRATCH;
where the $HOME version is linked to (at least in part). Note: There is no new instance of access-am3-configs/. Sorry, I never really acknowledged where (or how) the demarkation line was. I knew it was doing it, and it was just linked, but it just became muscle memory to know which version to go to.
Ahh I suspect I may know what is happening. Is this a configuration that was installed some time ago, and you’re reusing it for your test runs? And you’ve pulled the most recent version of dev-n96e? With CABLE changing to be included as a library, there are additional tasks in the graph to retrieve and build CABLE source.
The build_cable task is a dependency of the fcm_make_um task, but perhaps your task graph was not rebuilt, so it’s in a weird state where fcm_make_um thinks it needs build_cable to run, but build_cable is not in your task graph. Apparently, Cylc7 graphs don’t update when you reload, but Cylc8 ones do.
At some stage yesterday I was using an updated config. Not entirely sure, but I think this might’ve been the dev-n96e-vn13.8 branch and it did show a graph that looked more in line with sources coming from libraries etc.
That’s the expected graph for the default configuration, that doesn’t do the in-suite build. Do the install_ tasks ever finish? If there are issues with the directory Cylc thinks it should install to and the mounted directories, this can happen, but I don’t see any issues with that in your setup.
This time around the recon finished as well. Atmos ran until dt=525. But there are no (or sometimes empty) log files. I haven’t looked at all the pe_output yet to see where it failed
