Creating a regional ACCESS-OM3

COSIMA TWG
1

A new thread to help build a regional ACCESS-OM3 configuration using MOM6 and NUOCPY.

Progress so far.

Firstly you want to get Spack installed:

https://access-hive.org.au/getting_started/spack/

Then to build using Spack, you want to use these instructions
https://access-hive.org.au/models/run-a-model/build_a_model/

At this step:

git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5.git
spack env create mom5_dev ACCESS-ESM1.5/spack.yaml

Change it to

git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git
spack env create mom6_dev ACCESS-OM3/spack.yaml
#update: typo fixed thanks to Paul's feedback

Note, if you attempt the Test Spack (OPTIONAL) section, the concretize step will fail.

If you want to start editing the source code, move to spack develop (my intuition suggests that this will be based on the repo’ you used to create the environment), namely:

[cyb561.gadi-login-01: environments]$ spack env activate mom6_dev
[2007][cyb561.gadi-login-01: environments]$ spack find
#get name of the nuopc version you've already set yourself up with, in my case..
#make sure you do a module purge!
spack -d develop  access-om3-nuopc@git.0.3.1=0.3.1
#code for me ends up here
#/g/data/tm70/cyb561/spack/0.22/environments/mom6_dev/access-om3-nuopc
spack install

A successful install of the base MOM6 model will look like this

$ spack find
==> In environment mom6_dev
==> 1 root specs
[+] access-om3@git.2024.09.0=2024.09.0

==> Installed packages
-- linux-rocky8-x86_64 / intel@2021.10.0 ------------------------
access-om3@git.2024.09.0=2024.09.0  cmake@3.24.2    fortranxml@4.1.2  libaec@1.0.6     m4@1.4.19             parallelio@2.6.2  zstd@1.5.6
access-om3-nuopc@git.0.3.1=0.3.1    diffutils@3.10  glibc@2.28        libiconv@1.17    netcdf-c@4.9.2        pkgconf@2.2.0
bzip2@1.0.8                         esmf@8.5.0      gmake@4.4.1       libsigsegv@2.14  netcdf-fortran@4.6.1  snappy@1.1.10
c-blosc@1.21.5                      fms@2023.02     hdf5@1.14.3       lz4@1.9.4        openmpi@4.1.5         zlib-ng@2.1.6

To run an experiment, follow the guidelines here

https://access-hive.org.au/models/run-a-model/run-access-om/

You can execute the following in~/access-om3 to clone an experiment from branch dev-1deg_jra55do_ryf, and creating a local branch expt:

$ payu clone -b expt -B dev-1deg_jra55do_ryf https://github.com/ACCESS-NRI/access-om3-configs 1deg_jra55do_ryf

Cloned repository from https://github.com/ACCESS-NRI/access-om3-configs to directory: /home/548/pag548/access-om3/1deg_jra55do_ryf
Created and checked out new branch: expt
laboratory path:  /scratch/gb02/pag548/access-om3
binary path:  /scratch/gb02/pag548/access-om3/bin
input path:  /scratch/gb02/pag548/access-om3/input
work path:  /scratch/gb02/pag548/access-om3/work
archive path:  /scratch/gb02/pag548/access-om3/archive
Metadata and UUID generation is disabled. Experiment name used for archival: 1deg_jra55do_ryf
To change directory to control directory run:
  cd 1deg_jra55do_ryf

The config.yaml file for MOM6 is a bit simpler to MOM5. You only have to specific the location of a single executable. Find the location of your executable built using spack and assign it to the exe entry of your ~/access-om3/<branch-name>/config.yaml file, e.g.

model: access-om3

exe: /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.1_0.3.1-756wxghdm5i75v26j5oybrutennrjdyi/bin/access-om3-MOM6-CICE6

input:
    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc
    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc

etc.

Then run payu setup and check the following files exist in ~/access-om3/<branch-name>/manifests/
e.g.

$ ls ~/access-om3/1deg_jra55do_ryf/manifests/
exe.yaml  input.yaml  restart.yaml

Then run your experiment using payu run.

The manifest files are scanned by payu run to locate the gadi projects for all required inputs (including your executable) and create the correct -l storage flag passed to the the PBS qsub command.

3 Likes
2

Ok @navidcy , @PSpence , @Helen , @AndyHoggANU and @cbull we are continuing this thread now on the hive.

Question : Did we find the answer to the problem of payu not being able to find the executable? e.g. my config.yaml has

exe: /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.1_0.3.1-756wxghdm5i75v26j5oybrutennrjdyi/bin/access-om3-MOM6-CICE6

But payu cannot run the model:

FileNotFoundError: Executable for access-om3 model not found on path: /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64/intel-2021.1
0.0/access-om3-nuopc-git.0.3.1_0.3.1-756wxghdm5i75v26j5oybrutennrjdyi/bin/access-om3-MOM6-CICE6
3

Hi Paul, thanks for collating my instructions and your experience into this post from here, specifically 1, 2, 3, 4.

Question : Did we find the answer to the problem of payu not being able to find the executable?

Yeah, so, I suggest you try using my executable? (i.e. /g/data/tm70/cyb561/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.1_0.3.1-va553vmg3iv3i7uknyyymh34nna2hz7s/bin/access-om3-MOM6-CICE6 from here) It’s curious because @helen was able to use mine, so we expect there might be something wrong with the environment.

We discussed this at a standup last week (@minghangli @anton @ezhilsabareesh8), there were a few suggestions around your environment that can complicate:

  • when one has done their own pip install of payu that will get preferentially used over a module load although your payu version looks quite new.
  • worth looking at the standard output on payu run as this will tell you how the project code is being interpreted and whether you can expect to have access to your executable
  • worth trying a payu setup as that will look to see if it can find the executable (if this works one then needs to try a payu sweep, then payu run)
4

I don’t think Chris’ build will work for you as you aren’t a member of the tm70 project.

When you do a payu run it should print the qsub command out, the first thing is to check the -l storage = flag includes your project.

I think you’ll need to run payu setup manually, and then next time payu run the paths within the storage = flag will be updated.

1 Like
5

Ok running payu setup worked. The file manifests/exe.yaml now contains

format: yamanifest
version: 1.0
---
work/access-om3-MOM6-CICE6:
  fullpath: /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.1_0.3.1-756wxghdm5i75v26j5oybrutennrjdyi/bin/access-om3-MOM6-CICE6
  hashes:
    binhash: 59a6707be5e899664bbf06105362ff54
    md5: db3315a0e5580b5e8ad2b4b78dc9f185

and the -l storage flag for the PBS sub command created by payu contains gb02:

qsub -q normal -P gb02 -l walltime=01:00:00 -l ncpus=240 -l mem=960GB -l jobfs=10GB -N 1deg_jra55do_ry -l wd -j n -v PAYU_PATH=/g/data/vk83/apps/payu/1.1.5/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles  -l storage=gdata/gb02+gdata/vk83 -- /g/data/vk83/apps/payu/1.1.5/bin/python3.10 /g/data/vk83/apps/payu/1.1.5/bin/payu-run

So what’s next? Should I move to a spack development branch and run install again?

Just to clarify : I’ve just run ACCESS-OM3 using MOM6 coupled to CICE6 using NUOPCY on a global grid.

What’s the next step? Building the regional OM3 suite using NUOPCY?

2 Likes
6

Hi Paul

We are going to meet tomorrow anyway, but I did manage to hack together a configuration without CICE which runs. There are some more ice related config lines which can probably be removed.

Another thing to test is building without CICE, probably like this:

And then seeing if the resulting access-om3-MOM6 executable works.

7

Good to catch up and meet @anton, @cbull and @ashjbarnes yesterday.

I’ll try at get regional MOM6 up and running soon.

@MartinDix and @kieranricardo - can you tell me the suite-id for the CM3 (?) development suite that uses NUOPCY?

Thanks

8

It was good to catchup the other day – thanks @anton and @ashjbarnes for coming along to help us out. I am just putting us a few helpful links that came out of the meeting for our reference.

  1. A list of the changed needed to turn a global CESM into a regional CESM
    Make a CESM workflow ¡ Issue #1 ¡ CROCODILE-CESM/regional-mom6 ¡ GitHub

  2. NCAR’s “wrapper” for regional mom6, which implements the above checklist to automate the global → regional model conversion.
    GitHub - CROCODILE-CESM/regional-casegen at 300fbc1c87d022389ef11c67a8a54a5c9183da10

  3. Github repository of regional mom6. This will create a system-agnostic input files to run a regional mom-6 configuration, but extra files will be needed to run with nuopcy
    GitHub - COSIMA/regional-mom6: Automatic generation of regional configurations of the Modular Ocean Model 6 (MOM6) in Python

  4. COSIMA recipe for setting up a regional mom6 configuration using om2 model outputs. This is currently not with the nuopc coupler, but when we have something working on nuopc then we could modify this
    cosima-recipes/Recipes/regional-mom6-forced-by-access-om2.ipynb at main ¡ COSIMA/cosima-recipes ¡ GitHub

  5. Anton has built ACCESS-OM3 with mom-only and symmetric (something @ashjbarnes flagged that we needed to do)
    test a MOM6 only build of OM3 by anton-seaice ¡ Pull Request #30 ¡ ACCESS-NRI/ACCESS-OM3 ¡ GitHub

  6. A description of the input files:
    Configurations ¡ COSIMA/access-om3 Wiki ¡ GitHub

2 Likes
9

hey @Paul.Gregory, our CM3 suite is on GitHub - what’s your username? I’ll add you to the repo

10

Rush74

1 Like
11

Hi @anton, I am attempting to compile and run mom6-standalone as a global configuration before trying as regional. I am using the above links to your spack build pull request for compiling and then running using your configuration files. Thanks for pulling these together – they almost worked straight out of the box for me.

However, I did encounter a couple of issues and I would like to flag them with you in case they are user errors that I should sort out before I go too much further!

Issue 1
When I build spack using your mom-standalone version:

git clone -b mum6-nuopc-standalone-2 git@github.com:ACCESS-NRI/ACCESS-OM3.git
spack env mom6_standalone_dev create  ACCESS-OM3/spack.yaml
spack env activate -p mom6_standalone_dev
cd ACCESS-OM3
spack concretize -f –fresh

I get an error:

2024.12.mom6 is not a valid git ref for access-om3

I think this is similar to the error that Paul and I received when running this command in the spack test to check we had compiled correctly. I believe that this is reported and being delt with?

Instead, I used an already working version of OM3 spack build but included your modifications

i.e. in spack.yaml I added

- configurations=MOM6
- +mom_symmetric

under

access-om3-nuopc:
require:

I then followed the usually instructions for compiling with spack and this seems to work. I think I now have a symmetrical mom-only executable (but am unsure, given the next errors!). Should the two lines I added be enough to make it symmetric and mom-only?

Issue 2
At run time I am using your version of the configuration files, my newly compiled executable and payu to setup the experiment:

payu clone -b expt -B mom6-nuopc-standalone https://github.com/anton-seaice/MOM6-CICE6 mom6-nuopc
payu setup

I get this error from payu:

ntasks = int(self.runconfig.get(“PELAYOUT_attributes”, f"{realm}_ntasks"))

I think the issue is that payu is still expecting an “ntasks” for waves and ice in nuopc.runconfig so I added these lines back, but changed the numbers so no resources were given to ice
i.e. I added this into nuopc.runconfig

ice_ntasks = 1
ice_nthreads = 1
ice_pestride = 0
ice_rootpe = 0

I added something similar for waves. This now runs for a bit (using payu run), but I get some errorrs as it tries to output CICE variables. I am still investigating this but thought I should check in with what I have done to see if I need to fix up some earlier missteps first. Thanks!

12

Hi Helen

You are correct about Issue 2, we removed the requirement for these lines in https://github.com/payu-org/payu/pull/496 (as the model doesn’t need them). This would be solved in the next payu release, (or you can use the payu pre-release version module use /g/data/vk83/prerelease/modules; module load payu/dev ).

I have not got my head around Issue 1. The git ref for the spac spec for a deployment package (in this case access-om3) is the tag that would be created if that version was deployed. So I don’t understand how it can be “not a valid git ref” because it doesn’t exist yet.

If there is an ‘access-om3-MOM6’ executable available then it has built the MOM6 only version.

I think running spack find -v access-om3-nuopc should confirm if its built with symmetric memory (i.e. +mom_symmetric) or not (i.e. ~mom_symmetric)

13

Thanks @anton!

I will swap to payu/dev – so I think issue 2 is now not an issue.

For issue 1: I have an access-om3-MOM6 executable that is “+mom_symmertric” so things are looking good!

I believe that the “not a valid git ref” issue has appeared before and has been raised by Paul here:

14

Hey folks, psyched to see all this action! I’ve just started with 21st Century Weather and will be commencing my MOM6 journey with Ashley’s MOM6-R package. But I am lurking on this issue to follow your progress & learn in the hope of contributing soon!

2 Likes
15

Hi Madi - I am excited to hear that you are getting straight into this!

I have now (as of 2 days ago!) configured regional mom6 standalone to run on nuopc and I have run a 4-day test case. I am writing up some instructions on the steps I needed to do to get this which will hopefully be available in the next week or so. It would be great if you (and @Paul.Gregory?) had some time to follow my instructions and provide feedback. Step 1 will be something along the lines of “commence your MOM6 journey with Ashley’s MOM6-R package” – so I think this is a great place to get started.

1 Like
16

Hi Helen that’s incredible! I’d love to test it at any stage! let me know :smiley:

1 Like