I’ve been working to transfer some UMUI experiments from Accessdev to Gadi before Accessdev is decommissioned over the coming weeks, however have run into some issues after following the instructions. After some trial and error, the UMUI seems to go through the processing and submission steps and does not show any errors. Unfortunately the jobs don’t actually run, and instead the “umuisubmit_clr” script leaves behind the following errors:
Base build: failed
/local/spool/pbs/mom_priv/jobs/107267198.gadi-pbs.SC[10]: fcm: not found [No such file or directory]
The file that appears to cause the error in umuisubmit_clr ($UM_ROUTDIR/umbase/cfg/bld.cfg) exists when I follow the path, and so I’m unsure what’s causing the error.
I was just hoping to check whether anyone else has still been using the UMUI models (specifically version 7.3), and might have run into similar issues?
Thank you for having a look into this! That’s a good point about fcm. I’ve been loading in fcm before opening up the UMUI, but the UMUI ssh’s into localhost when submitting the run and so I’m guessing that the modules don’t stay loaded when it does that.
Interestingly copies of umuisubmit_clr from runs I’ve submitted from ACCESSdev start of by loading in the modules and specifying the PBS directives:
but the copies of the same script from the runs I’ve tried to submit from gadi don’t try to load in any of the modules and don’t set any of the PBS directives:
I am not completely aware of the workflow for submitting UMUI runs from Gadi.
There may be a pre-script where all the module loads are carried out.
However, to check if the problem is the fcm module (as it seems to be), I would add a couple of lines line at the beginning of the umuisubmit_clr script (after all the export lines), something like:
cat <<EOF
++++++++++++++++++++++++++++++
START FCM TEST
EOF
fcm --version
cat <<EOF
END FCM TEST
++++++++++++++++++++++++++++++
Then submit the script again, and check the output.
I think you will get the same error, but you will still get the output:
++++++++++++++++++++++++++++++
START FCM TEST
(Note you should check this output in the output log file, not in the error log file).
In that case, you can try removing the previously added lines and, instead, add a line where you import the fcm module:
module load fcm
In this case, if it still fails, it should be with a different error.
Let me know how it goes.
Thank you for your suggestions with this, and apologies in the delay in getting back to you!
I’ve done a bit more digging and have been able to get the model running. As you suggested, the issue was caused by the submission scripts not loading in fcm.
I wasn’t able to edit the umuisubmit_clr script directly, as the UMUI generates it and runs it all in one go when you submit a simulation. After some digging though, the difference between the gadi and accessdev versions of umuisubmit_clr appear come from differences in another script called SUBMIT. SUBMIT is generated when you click “process” in the UMUI, and it’s used later to generate the other submission scripts when you click “submit” in the UMUI.
The SUBMIT script from an Accessdev run includes all the PBS and module instructions to paste into umuisubmit_clr:
There’s a file located on Accessdev at /projects/access/gadi/inserts/um7.3/compile.pbs which contains these instructions, and so I’m wondering whether it’s just that the UMUI on gadi doesn’t look for or find it. I’ve attached it below for future reference.
Anyway, the workaround that ended up being ok was to replace the gadi SUBMIT script with the accessdev one (attached) and adjusting its experiment ID settings in between the processing and submission steps. I’m surprised it actually worked but am glad it did!
For the future, you can still edit the umuisubmit_clr script (created by UMUI), and then submit it by running:
qsub ~/path/to/the/umuisubmit_clr
This way, you don’t necessarily have to set up your experiment from UMUI (for testing for example).
Yes, all umuisubmit scripts are created from the SUBMIT script (you could even run the SUBMIT script yourself to generate the umuisubmit scripts)
Anyway, I am happy to know you found a solution for your experiment.
Also know that there is the option to add top/bottom scripts run at the beginning/end of your experiment run.
These could be used, for example, to load any modules you require in your experiment.
You can add these scripts in UMUI > edit experiment > Input/Output Control and Resources > Scripts Inserts and Modifications
In the panel click “Using bottom and top script inserts” and you will be able to insert the directory and path to your top/bottom scripts.