CESM configurations on Gadi using CIME

CIME (pronounced “SEAM”) is a Python package for configuring, compiling and running earth system models. It is used by a number of models, including CESM, UFS, HAFS, E3SM and NorESM. Kieran Ricardo and I are currently using CIME on Gadi to run and explore some CESM model configurations that could be helpful for developing the next generation of ACCESS models.

To get CIME working on Gadi, some Gadi specific configuration has been added to forks of various repos. These forks live in the ACCESS-NRI Github org. You can clone the ACCESS-NRI CESM fork and dependent repos using:

git clone -b  access-nri-repos git@github.com:ACCESS-NRI/CESM.git
cd CESM
./manage_externals/checkout_externals -v -o

This will give you a copy of CESM and CIME to build and run it. For example, to run a CESM model case using DATM, MOM6, CICE6 and SWAV, coupled using NUOPC/CMEPS (note this works best if your default PROJECT is tm70 so that you can use a shared input data directory):

cd cime/scripts
./create_newcase --case test --compset GMOM_JRA --res T62_g16 --machine gadi
cd test
./case.setup
./case.build
./case.submit

(Currently, this case will just execute each model component sequentially on a single node)

There is also an officially unsupported model configuration using DATM, MOM6, CICE6 and WW3 (i.e. essentially ACCESS-OM3), however I’m currently trying to track down some missing input files to get this running.

CESM configurations that use NUOPC/CMEPS can also be run from payu. For now, the payu CESM model driver only exists in a branch in my fork of payu, so anyone wanting to try this out will have to clone and install that.

Payu configs for a couple of CESM configurations can be found at:

Be aware, these are untested payu implementations of officially-unsupported CESM configurations. You’re very welcome to give them a spin, but do so at your own risk :slight_smile:

I’ve also kluged together a CICE6-WW3-DOCN-DATM-DROF CESM executable. Payu config can be found at GitHub - dougiesquire/d_jra_wd: Kluged-together CESM CICE6-WW3-DOCN-DATM-DROF-SLND-SGLC build scripts and payu configuration.

Same warning as above, but even more so since this configuration doesn’t even exist as a compset in CIME.

cc @sofarrell

Hi @dougiesquire. I am new to the CESM community. I am following your above instructions to run a test case. When running “./case.setup”, it wants to create folders (specifically inputdata) in the directory “/g/data/hh5” instead of my /g/data/w40/user directory in which I cloned CESM and ran “.create_newcase”. Do you perhaps know how to redirect this to the correct directory?

@Michael_Barnes are you cloning CESM from the ACCESS-NRI org as above? You should be able to change the inputdata location by editing DIN_LOC_ROOT for the "gadi" machine in ccs_config/machines/config_machines.xml (line 3651)

I should emphasise that the ACCESS-NRI forks of CESM and associated repos are not maintained or supported and haven’t been updated in over a year. These were only set up to explore CESM configurations on which the next generation of ACCESS models have been based.

Hi @dougiesquire. Thanks for the info. Yes, I cloned the ACCESS-NRI one. Do you think I should clone CESM directly from UCAR and add in the machine manually rather than cloning the ACCESS-NRI one?

That will probably make things easiest for you to manage/update going forward. You’ll need to make a few changes across a couple of repos to get things working. I’d suggest:

  • Create your own fork of ccs_config_cesm. Copy the changes made here. Note, it looks like the structure of this repo has changed since those changes were made, but hopefully it’s clear what goes where.
  • Create your own fork of MOM_interface. Copy the changes made here. Note, no thought has been put into the PE layout on Gadi. Layout specifications were added for the MOM + "DATM.+CICE.+WW3" compset, but this just allocates each component 1 node and runs them sequentially. Running other compsets on Gadi will use whatever the default specification is for that compset. Hopefully the changes in the ACCESS-NRI fork on MOM_interface at least show how you would configure PE layout for specific compsets if/when you need to.
  • Create your own fork of CESM. Copy the changes made here but instead use the forks you made above. You don’t need to worry about the ParallelIO repo as the changes we made are now in the main repo.

Hopefully you can then follow the instructions in this topic, but cloneing your own CESM fork, rather than the ACCESS-NRI one

Thanks so much @dougiesquire!!! I will give this a go.