Introduction
These are the steps needed to convert an ACCESS-OM3 model configuration to a regional model configuration run on NUPOC.
If you would like to view an example setup there is one here. To download this you need to run
git clone --branch dev-regional_jra55do_ryf https://github.com/ACCESS-NRI/access-om3-configs/
if you want to see some of steps described in the instructions below then run
cd access-om3-configs
git log -16
This will provide a log of the changes performed to convert the original ACCESS-OM3 run to an ACCESS-ROM3 run. The output will display a long alphanumeric commit hash for each step.
To see the changes done to the code between two different steps you can use the first few digits of the commit hashes for each of these steps.
e.g
git diff ba62 a6b7
The following instructions are the steps needed to generate a regional ocean model (ACCESS-ROM3). Keep an eye on these instructions as they will be modified over the next few weeks for bug fixes and changes in the underlying code.
Compile
The first step is optional as there is a version of the executable already available to be used.
I suggest skipping this step for now and waiting a couple of weeks before trying as there are some changes coming through that will make it easier.
To skip this step, go to the next step: Generate your input files
Get a Spack build using mom-only symmetric on gadi by first installing spack instructions here. Note that you might get an error when doing the spack concretize step. It is not an issue here so ignore It.
Next you need build spack using a modification of instructions here
The modifications are when you reach this step:
git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5.git
spack env create mom5_dev ACCESS-ESM1.5/spack.yaml
you need to change to this:
git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git
spack env create mom6_dev ACCESS-OM3/spack.yaml
If you wait a couple of weeks, these next changes will not be necessary
Then, in spack.yaml you need to change
access-om3-nuopc:
require:
- '@git.0.3.1'
to
access-om3-nuopc:
require:
- '@git.0.3.1'
- configurations=MOM6
- +mom_symmetric
Generate your input files
Use regional-mom6 notebook from the cosima-recipes to get a non-nuopc regional mom 6 configuration.
There needs to be some modifications to change the date_range to [“2013-01-01”, “2013-01-05”] and to change the dataset used.
change
experiment = catalog["01deg_jra55v13_ryf9091"]
to
experiment = catalog["01deg_jra55v150_iaf_cycle1"]
This is good for getting something up and running – but when doing production runs, you should carefully consider which model run is best suited for boundary conditions.
There is an example of the notebook with the modifications here
Note that this example uses the crocodile version of regional-mom6. It should work on the COSIMA version in a few weeks after a pull request has gone through.
Remember which folder you put these files in as you will need to change some files later to point to it. i.e in the notebook we have:
input_dir = f"{scratch}/regional_mom6_configs/{expt_name}/"
run_dir = f"{home}/mom6_rundirs/{expt_name}/"
These folders are going to be called “input_dir” and “run_dir” throughout the rest of these instructions.
This generates some of the input files that we need, but the nuopc files need to come from a global configuration.
Download your other configuration files from an ACCESS_OM3 run
Download a global access-om3 configuration onto gadi. Do this in a different folder than the input_dir that you used in the notebook in a previous step
mkdir -p ~/access-om3
cd ~/access-om3
module use /g/data/vk83/modules
module load payu/1.1.6
payu clone -b expt -B dev-1deg_jra55do_iaf https://github.com/ACCESS-NRI/access-om3-configs/ access-rom3
cd access-rom3
We now need to mesh our regional mom simulation which has our input files (input_dir) and the global configuration that runs on nuopc.
Copy the MOM_input, MOM_override, MOM_layout and data_table files from run_dir to ~/access-om3/access-rom3/
Modify MOM_input to point to different locations
There needs to be quite a few changes to the MOM_input file
First changes are to point to different locations for the input. Note that the changes in this step will not be needed in future versions of regional-mom6 toolbox. If you cannot find these lines to change in MOM_input then hopefully you can skip this step (still do the last atmospheric forcing change in the next post) but watch for errors about missing boundary files.
Change
TEMP_SALT_Z_INIT_FILE = "forcing/init_tracers.nc" ! default = "temp_salt_z.nc"
to
TEMP_SALT_Z_INIT_FILE = "init_tracers.nc" ! default = "temp_salt_z.nc"
Change
SURFACE_HEIGHT_IC_FILE = "forcing/init_eta.nc" !
to
SURFACE_HEIGHT_IC_FILE = "init_eta.nc" !
Change
VELOCITY_FILE = "forcing/init_vel.nc" !
to
VELOCITY_FILE = "init_vel.nc" !
Change
OBC_SEGMENT_001_DATA = "U=file:forcing/forcing_obc_segment_001.nc(u),V=file:forcing/forcing_obc_segment_001.nc(v),SSH=file:forcing/forcing_obc_segment_001.nc(eta),TEMP=file:forcing/forcing_obc_segment_001.nc(temp),SALT=file:forcing/forcing_obc_segment_001.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_002_DATA = "U=file:forcing/forcing_obc_segment_002.nc(u),V=file:forcing/forcing_obc_segment_002.nc(v),SSH=file:forcing/forcing_obc_segment_002.nc(eta),TEMP=file:forcing/forcing_obc_segment_002.nc(temp),SALT=file:forcing/forcing_obc_segment_002.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_003_DATA = "U=file:forcing/forcing_obc_segment_003.nc(u),V=file:forcing/forcing_obc_segment_003.nc(v),SSH=file:forcing/forcing_obc_segment_003.nc(eta),TEMP=file:forcing/forcing_obc_segment_003.nc(temp),SALT=file:forcing/forcing_obc_segment_003.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_004_DATA = "U=file:forcing/forcing_obc_segment_004.nc(u),V=file:forcing/forcing_obc_segment_004.nc(v),SSH=file:forcing/forcing_obc_segment_004.nc(eta),TEMP=file:forcing/forcing_obc_segment_004.nc(temp),SALT=file:forcing/forcing_obc_segment_004.nc(salt)" !
! OBC segment docs
to
OBC_SEGMENT_001_DATA = "U=file:forcing_obc_segment_001.nc(u),V=file:forcing_obc_segment_001.nc(v),SSH=file:forcing_obc_segment_001.nc(eta),TEMP=file:forcing_obc_segment_001.nc(temp),SALT=file:forcing_obc_segment_001.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_002_DATA = "U=file:forcing_obc_segment_002.nc(u),V=file:forcing_obc_segment_002.nc(v),SSH=file:forcing_obc_segment_002.nc(eta),TEMP=file:forcing_obc_segment_002.nc(temp),SALT=file:forcing_obc_segment_002.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_003_DATA = "U=file:forcing_obc_segment_003.nc(u),V=file:forcing_obc_segment_003.nc(v),SSH=file:forcing_obc_segment_003.nc(eta),TEMP=file:forcing_obc_segment_003.nc(temp),SALT=file:forcing_obc_segment_003.nc(salt)" !
! OBC segment docs
OBC_SEGMENT_004_DATA = "U=file:forcing_obc_segment_004.nc(u),V=file:forcing_obc_segment_004.nc(v),SSH=file:forcing_obc_segment_004.nc(eta),TEMP=file:forcing_obc_segment_004.nc(temp),SALT=file:forcing_obc_segment_004.nc(salt)" !
! OBC segment docs
Changes to MOM_input to put NUOPC atmospheric forcing info
This change to MOM_input will still need to be done with future versions of regional mom6-toolbox. We need to add some lines for atmospheric forcing coming in via nuopc coupler:
Add this to the bottom of MOM_input:
! === module MOM_surface_forcing_nuopc ===
ENTHALPY_FROM_COUPLER = True
! "[Boolean] default = False
! If True, the heat (enthalpy) associated with mass entering/leaving
! the ocean is provided via coupler."
LATENT_HEAT_FUSION = 3.337E+05 ! [J/kg] default = 3.337E+05
LATENT_HEAT_VAPORIZATION = 2.501E+06 ! [J/kg] default = 2.501E+06
Changes to config.yaml to point to input netcdf
You also need to make some changes in the config.yaml file under “input:” to add these lines make sure you change “input_dir” and “run_dir” to your input run directories created in the regional-mom6 notebook:
- input_dir/hgrid.nc
- input_dir/vcoord.nc
- input_dir/bathymetry.nc
- input_dir/init_tracers.nc
- input_dir/init_eta.nc
- input_dir/init_vel.nc
- input_dir/forcing_obc_segment_001.nc
- input_dir/forcing_obc_segment_002.nc
- input_dir/forcing_obc_segment_003.nc
- input_dir/ forcing_obc_segment_004.nc
- input_dir/grid_spec.nc
- input_dir/ocean_mosaic.nc
- input_dir/access-rom3-ESMFmesh.nc
- input_dir/access-rom3-nomask-ESMFmesh.nc
- input_dir/land_mask.nc
Note that the .mesh.nc files do not yet exist.
I would suggest double checking that the above links (except for *.mesh.nc) are correct as the regional-mom6 toolbox is under active development and the position of these files will change depending on which version you are using. In particular, some of the files may be under input_dir/forcing.
You will need to remove these lines:
- /g/data/vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
- /g/data/vk83/configurations/inputs/access-om3/mom/grids/vertical/global.1deg/2023.07.28/ocean_vgrid.nc
- /g/data/vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc
- /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc
- /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc
if you are using an executable compiled by ACCESS-NRI then you also need to change this:
load:
- access-om3/2025.01.1
to this:
load:
- access-om3/2025.01.0
Changes to config.yaml to point to executable
Note, if you compiled your own executable that you need to skip this step and go to alternative changes to config.yaml These changes won’t be needed in a few weeks’ time.
Change
exe: access-om3-MOM6-CICE6
to
exe: access-om3-MOM6
Alternative changes to config.yaml
If you have compiled your own executable then you need to point your executable to the one you created using spack. You do not need to do this step if you did the previous step
Change this
exe: access-om3-MOM6-CICE6
to something that looks like this – but your path will vary
exe: /g/data/***/****/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.1_0.3.1-6ry27bdy6uoet34nwk6yunmkrkomvads/bin/access-om3-MOM6
Changes to config.yaml job specifications
You also need to modify these lines in config.yaml:
-
ncpu: 240
This is the number of CPUs you will need. It depends on the layout that you specified when calloing FRE_tools. If you didn’t change the default (10x10) in the regional-mom6 notebook then you want
ncpu:100 -
change
runlog: falsetorunlog: true -
mem:960GB
The number you need here will be domain dependent. For the Tasmanian example I use
mem:100GB
You may need more when using a larger domain -
Comment out these line for mom-standalone runs
#userscripts:
# setup: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/setup.sh
# archive: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/archive.sh
This last step is setting up CICE so keep it in for simulations with ice.
Creating mesh files
You will need to use generate_mesh.py from om3-scripts
To do this you need to download a copy of the toolbox. Note that in the following commands you need to change “path_to_where_you_store_libraries” to wherever you want to download this to.
cd path_to_where_you_store_libraries
git clone https://github.com/COSIMA/om3-scripts.git
conda activate analysis3
The following commands will create your mesh files. Note that you will need to change “input_dir” to your input_dir and “path_to_where_you_store_libraries” to where you just downloaded the om3-scripts to.
cd input_dir
python3 path_to_where_you_store_libraries/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=hgrid.nc --mesh-filename=access-rom3-ESMFmesh.nc --mask-filename=ocean_mask.nc --wrap-lons
python3 path_to_where_you_store_libraries/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=hgrid.nc --mesh-filename=access-rom3-nomask-ESMFmesh.nc --wrap-lons
Run these commands the input_dir so the files are stored with your other netcdf files. If you store them somewhere else, note that you will need to change the path to them in config.yaml
You need to modify a few configuration files to point to these. Note that the path we are putting in is different to their current location but payu will fix this up at run time.
Change back into your access-rom3 directory
cd ~/access-om3/access-rom3
In datm_in change these lines:
model_maskfile = "./INPUT/access-om2-1deg-nomask-ESMFmesh.nc"
model_meshfile = "./INPUT/access-om2-1deg-nomask-ESMFmesh.nc"
nx_global = 360
ny_global = 300
to
model_maskfile = "./INPUT/access-rom3-nomask-ESMFmesh.nc"
model_meshfile = "./INPUT/access-rom3-nomask-ESMFmesh.nc"
nx_global = 140
ny_global = 249
Note that nx_global and ny_global are the domain size. The numbers used here are from the Tasmanian example but if you have a different size domain then you can check these numbers in MOM_layout (the value for NIGLOBAL is the same as nx_global and the values for NJGLOBAL are the same as ny_global).
Similar, you need to modify drof_in to point to the masks
Change
model_maskfile = "./INPUT/access-om2-1deg-nomask-ESMFmesh.nc"
model_meshfile = "./INPUT/access-om2-1deg-nomask-ESMFmesh.nc"
nx_global = 360
ny_global = 300
to
model_maskfile = "./INPUT/access-rom3-nomask-ESMFmesh.nc"
model_meshfile = "./INPUT/access-rom3-nomask-ESMFmesh.nc"
nx_global = 140
ny_global = 249
Changes to input.nml
In input.nml you need to include an extra file.
Change
parameter_filename = 'MOM_input', 'MOM_override'
to
parameter_filename = 'MOM_input', 'MOM_layout', 'MOM_override'
Changes to nuopc.runconfig for new layout
We need to change the distribution of tasks (assuming the default 10x10 layout and 140x249 grid size):
Change
ocn_ntasks = 216
to
ocn_ntasks = 100
Change
ocn_rootpe = 24
to
ocn_rootpe = 0
Change
ocn_nx = 360
ocn_ny = 300
to
ocn_nx = 140
ocn_ny = 249
if you are not using a 140x249 grid size (e.g. the Tasmanian example) then you can find ocn_nx and ocn_ny in MOM_layout (NIGLOBAL and NJGLOBAL respectively)
We need to set the start date to match the date done in the notebook and to change the length of the simulation to just a couple of days.
Change
start_ymd = 19000101
stop_n = 1
stop_option = nmonths
to
start_ymd = 20130101
stop_n = 2
stop_option = ndays
As we are running for 2 days we need to change the frequency of the restart files:
Change
restart_n = 1
restart_option = nmonths
to
restart_n = 2
restart_option = ndays
We need to update the meshes in here as well.
Change
mesh_mask = ./INPUT/access-om2-1deg-ESMFmesh.nc
mesh_ocn = ./INPUT/access-om2-1deg-ESMFmesh.nc
to
mesh_mask = ./INPUT/access-rom3-ESMFmesh.nc
mesh_ocn = ./INPUT/access-rom3-ESMFmesh.nc
For mom-standalong runs, remove the ice component.
Change
component_list: MED ATM ICE OCN ROF
to
component_list: MED ATM OCN ROF
Change
ICE_model = cice
to
ICE_model = sice
Changes to nuopc.runseq to remove CICE
For mom-only runs in nuopc.runseq remove these lines (to not run the ice model; essentially remove any line that has “ice” in it)
MED med_phases_prep_ice
MED -> ICE :remapMethod=redist
ICE
MED med_phases_diag_ice_ice2med
ICE -> MED :remapMethod=redist
MED med_phases_post_ice
MED med_phases_diag_ice_med2ice
Changes to diag_table to output more frequently
The default option was to output monthly. As the test run does not run for a month, we are swapping to daily output.
In diag_table change
"access-om3.mom6.h.rho2%4yr-%2mo", 1, "months", 1, "days", "time", 1, "months"
"access-om3.mom6.h.native%4yr-%2mo", 1, "months", 1, "days", "time", 1, "months"
"access-om3.mom6.h.z%4yr-%2mo", 1, "months", 1, "days", "time", 1, "months"
"access-om3.mom6.h.sfc%4yr-%2mo", 1, "months", 1, "days", "time", 1, "months"
To
"access-om3.mom6.h.rho2%4yr-%2mo", 1, "days", 1, "days", "time", 1, "months"
"access-om3.mom6.h.native%4yr-%2mo", 1, "days", 1, "days", "time", 1, "months"
"access-om3.mom6.h.z%4yr-%2mo", 1, "days", 1, "days", "time", 1, "months"
"access-om3.mom6.h.sfc%4yr-%2mo", 1, "days", 1, "days", "time", 1, "months"
Run the model
To run you need to use payu:
module use /g/data/vk83/modules
module load payu/1.1.6
payu setup
payu sweep
payu run
Note: That there is still a fix to come for a known issue with land runoff so don’t do any production runs just yet
