Thanks a lot for providing this estimate. It sounds like you could potentially increase the number of years per job. E.g. 5 or 10 years per submission might be feasible. I don’t know whether that would help to get more years through the queue.
I asked similar questions during a recent meeting with @atteggiani . I think he is working on these things. Would be great to keep in touch with progress on this.
This would be expected to increase performance since you spend less time queuing. For example, with ACCESS-OM2-1 we typically run 10 years in a single job submission. Depending on the number of processors there is different limits on the walltime (see the last column in Queue Limits - NCI Help - Opus - NCI Confluence). Although also keep in mind that if you make your individual submissions really long and you get a random error in the middle then you have to start the whole run again. You may also want to think about how your output files are saved (i.e. if you’re saving only one file per submission they might get big).
1 Like
@rmholmes Running 10 years in a single job should not take longer than 9 hrs for the N48 coupled suite. This estimated time is far less than the default wall time limit for 480 CPUs. We generally run 10-15 years in a single job for the N48 atmosphere-only version and want to follow the same in the coupled N48 suite. We believe this will enhance the effective computational performance of the model.
1 Like
FYI we added the runspersub option to payu for models which either couldn’t, or didn’t want, to run for longer, but weren’t effectively using the max wall timeavailable
https://payu.readthedocs.io/en/latest/config.html?highlight=runspersub#miscellaneous
Does the N48 model still use UMUI to run?
@Aidan The coupled N48 suite is based on rose/cylc.
1 Like
Great, thanks.
This depends on the model/scheduler you are using:
PAYU
See @Aidan’s answer above:
FYI we added the
runspersuboption topayufor models which either couldn’t, or didn’t want, to run for longer, but weren’t effectively using the max wall timeavailablehttps://payu.readthedocs.io/en/latest/config.html?highlight=runspersub#miscellaneous
UMUI
This can be done in Input/Output Control & Resources → Job submission, resources and re-submission pattern
At the bottom of the first pane there is a NEXT button, click on that and a new pane will open up.
You can “set the target run length for each job in the sequence” in the respective section.
Note that you might need to update your Job time limit (seconds per resubmission) and Job Memory limit in the first pane.
Rose/Cylc
In the cylc editor (rose edit -C <path_to_the_suite_folder>) under suite conf → Run Initialisation and Cycling you can change the Cylcing frequency (e.g. P10Y for 10 years re-submission chunks).
Here too, note that you might neet to change the Wallclock time as well (hh:mm:ss format)
Hope this helps.
Davide
1 Like
@atteggiani I can change the run length per job in UMUIX and rose/cylc using the options that you mentioned above. But the rose/cylc suite crashes if the run length in a single queued job is > 2.5 yrs or so. @holger told me a critical number related to model timesteps and other factors for the suite. The suite doesn’t run if the number exceeds, so the run length per job is kept to 2 years.
@holger could you please remind me of the critical value and how you got it?
If I remember correctly, OASIS reads the length of the run in seconds from its configuration file as an 8-digit integer, and 99,999,999 seconds is a bit over 3 years. If you try to run the job longer (at least with umuix) what happened was the namcouple file was written with 9-digit number, but only the first 8 digits were read in, so oasis believed the job to be finished after only a tenth of the job in.
1 Like
Hi, after some discussions with @Scott and @atteggiani we now think the best way forward is the following:
(1) The main problem is to change a land to an ocean point and an ocean point to land. We know how to do it in the UM model, but we do not know how to do it in the coupled model or ocean model.
(2) We should first change the land-sea distribution (e…g any simple change like closing Drake Passage and opening the Panama passage) in the ocean only run. Therefore we use an ocean only run that uses the some configuration as the coupled ACCESS-N48 model.
So our questions would now be @atteggiani @rmholmes @aekiss:
(1) What ocean only run can we use?
(2) What is the procedure to change the land-sea mask boundary conditions (potentially initial values and forcings)?
Best regards
Dietmar
1 Like
This section of the ACCESS-OM2 wiki describes how to change the bathymetry, land-sea mask and OASIS remapping weights in an ocean-only ACCESS-OM2 run.
With regards to (1) → I’m not sure as I haven’t worked with the coupled model. I imagine this configuration is a good starting point, but someone else probably knows better.
3 Likes
To turn an ocean grid cell into a land point (from the ocean model point of view): the main things you need to do are
(1) edit the topog.nc, i.e. set an exisiting non-zero depth cell to zero.
(2) Re-run a tool that creates the grid_spec.nc input file based on the edited topog.nc.
I have outlined a procedure to do this for the GFDL CM2.1 model, in step 1 of this wiki. Since ACCESS-ESM uses the same ocean, the procedure is basically the same (for the ocean model side). You will just have to use the ACCESS-ESM grid input files rather than the ones I used in my example (it’s a different resolution).
The slight complication here is that MOM5 can accept the grid_spec.nc input file using the “old method”, where all grid variables are included in grid_spec.nc, or the “new method”, where grid_spec.nc specifies the grid files through mosaic files that point to individual grid inputs for each model component.
I have noticed that ACCESS-ESM1.5 uses the “old method”… whereas the tool I specified creates these inputs using the “new method”, i.e. using mosaic input files. It is entirely possible to go between the two methods of creating grid_spec.nc, and I’m endeavouring to do this as a test case. I am going to try to do that… will report back to this post when I have more info.
1 Like
Dear @Dietmar_Dommenget,
This ACCESS-OM2 configuration (as pointed out by @rmholmes) is your best choice.
In particular, you could start running the 1 degree ocean ‘repeated-year-forcing’ (ryf) control configuration found here.
For more information on how to run it, check this quick-start guide.
For how to change land/ocean points and generate the respective associated land-sea masks (mapped to the atmospheric grid and mapped to the ocean grid), as well as the coupling weights needed by the coupler for the conversion between the two grids, you can start by modifying your bathymetry file (topog.nc) in a fashion similar to the one described by @dkhutch (any non-zero value will be attributed to ocean, and any value equals to 0 will be attributed to land).
Then you can follow this guide (also mentioned by @rmholmes) for the following steps.
In case you need help, please let us know in this post and I am sure someone will be able to give you a more direct support.
Cheers
Davide
Dear Ryan,
I tried to follow your tutorial today and got an error message at step " 2. Changing the associated land–sea masks":
python topog2mask.py topog.nc kmt.nc ocean_mask.nc
Traceback (most recent call last):
File “/g/data/w40/dxd565/access-om/experiments/no-drake-passage/input/…/…/…/tools/topogtools/topog2mask.py”, line 63, in
sys.exit(main())
File “/g/data/w40/dxd565/access-om/experiments/no-drake-passage/input/…/…/…/tools/topogtools/topog2mask.py”, line 59, in main
make_mask(args.topog, args.cice_mask, args.fraction, ice=True)
File “/g/data/w40/dxd565/access-om/experiments/no-drake-passage/input/…/…/…/tools/topogtools/topog2mask.py”, line 29, in make_mask
mask[:] = np.where((tf.variables[‘frac’][:] < frac) | (tf.variables[‘depth’][:] <= 0.0), 0, 1)
KeyError: ‘frac’
Any idea what is wrong?
I also wonder: there are a number of files as ocean input:
/g/data/ik11/inputs/access-om2/input_236a3011/mom_1deg/
e.g. geothermal_heating.nc, etc.
Do I also need to change all of those?
best regards
Dietmar
Hi Dietmar,
It looks like Micael made a change to GitHub - COSIMA/topogtools: Tools related to changing ocean model topography and regenerating dependent model inputs. in late 2022 (post when I wrote that tutorial, which was quite a while ago) that has caused this. Specifically, he added a frac input to the topog2mask.py code (Add two new tools: one automatically fixes non-advective cells found … · COSIMA/topogtools@03cfc66 · GitHub) that seems to have broken backward compatibility. I’ll put an issue on that repository.
In the mean time, I think you can most likely fix this by just using an older version of the code. Can you try again checking out an older commit - specifically, in the git directory that you created when you did git clone https://github.com/COSIMA/topogtools.git try doing
git checkout 952cbb16ba7cf8cf55c4aad425f33e2edeb62fe5
and then run it again.
With regards to the second question: It depends on what you’re doing (i.e. do you want geothermal heating to change, although I’m pretty sure geothermal heating is not on in ACCESS-OM2 anyway)? However, if you’re not worried about these things and all you want to change is the topography and land-sea mask, then I don’t think so. Those files aren’t masked and the values look reasonable:
I hope that helps. My memory of that process is a bit fuzzy, but happy to try and help if you run into any other issues.
3 Likes
Dear Ryan,
I am now stuck at your step “3. Changing the OASIS remapping files”. Honestly I do not understand much of what you have describe there:
(1) Where is the script “make_remap_weights.py” and folder “access-om2/tools/”? I am not sure where I would look for this.
(2) I do not understand your command: “…/…/tools/make_remap_weights.py ./ /g/data/ik11/inputs/JRA-55/RYF/v1-4/ …/yatm_1deg/ --atm JRA55 --ocean MOM1 --npes 16”
Where does the information from my experiment go in here? I have a folder: /g/data3/w40/dxd565/access-om/experiments/no-drake-passage/ancil
This contains the files I just created from the previous steps (kmt.nc ocean_mask.nc topog.nc). I guess I need to use those?
Maybe if you have some time, we could go through this together in a video meeting?
best regards
Dietmar
Sorry for the confusion.
make_remap_weights.py and access-om2/tools/ are in the main access-om2 repository. Did you do a git clone https://github.com/COSIMA/access-om2?
Have you already run an ACCESS-OM2 simulation without any changes? That would be a good first step to make sure everything is working and you know how to run the ocean-only case.
Yes I’m happy to chat. I’m free all afternoon today - just send me an email.
I do not have your email address. You you email me! how about 2pm today?
You asked me to review a J Climate recently, so you should! 
. Just sent you an invite.