Building ACCESS-OM3 with WW3: Using Spack for Local Modifications of WW3

COSIMA Waves
1

Introduction

This guide explains the steps to build ACCESS-OM3 2025.01.0 (0.4.0) from source using Spack, with a specific focus on modifying and rebuilding WAVEWATCH III (WW3). If you need to make local modifications to WW3—such as tuning wave physics, or implementing new parameterizations—this guide will walk you through the process of building ACCESS-OM3 with your changes applied to WW3.

Note: These instructions are only for users modifying the ACCESS-OM3’s WW3 source code. If you want to use a prebuilt version, refer to the documentation here

Prerequisites

Before building ACCESS-OM3 with modifications to WW3, you need to set up Spack on your system. Follow these steps documented here to install and configure Spack on Gadi.

Setting Up a Spack Env">Setting Up a Spack Development Environment for ACCESS-OM3 with WW3

The following steps assume you are already inside the Spack installation directory:

cd /g/data/$PROJECT/$USER/spack/0.22

1. Create a Development Directory for Spack

First, create a new directory to manage your Spack development environment:

mkdir spack-dev
cd spack-dev/

2. Clone the ACCESS-OM3 Spack Configuration Repository

Clone the ACCESS-OM3 repository that contains Spack configuration files:

git clone git@github.com:ACCESS-NRI/ACCESS-OM3.git

:warning: Note: This repository contains the Spack environment configuration for ACCESS-OM3, not the actual model source code.

3. Create a Separate Directory for WW3 Development

Since we are focusing on modifying and rebuilding WW3, create a dedicated directory for WW3 development:

mkdir ww3-dev

4. Create and Activate a Spack Environment

Use the spack.yaml file from the cloned ACCESS-OM3 repository to create a new Spack environment:

spack env create -d . ../ACCESS-OM3/spack.yaml
spack env activate .

Modifying and Building ACCESS-OM3 with a Custom WW3 Fork

After setting up the Spack environment, follow these steps to modify and build ACCESS-OM3 with your own WW3 fork.

1. Clone the ACCESS-OM3 Model Repository

Next, clone the ACCESS-OM3 model source code (which includes MOM6, CICE6, and WW3) into your working directory:

git clone git@github.com:COSIMA/access-om3.git access-om3-nuopc  
cd access-om3-nuopc/

2. Update the WW3 Component to Use Your Fork

  1. Create a fork of the official WW3 repository:

  2. Edit .gitmodules to point to your fork instead of the default WW3 repository:

    vi .gitmodules

    Inside .gitmodules, locate the WW3 submodule entry and update the url to point to your fork, e.g.:

    [submodule "components/WW3"]
    path = components/WW3
    url = git@github.com:<your-github-username>/WW3.git

  3. Save and close the file.

3. Initialize and Update Submodules

After modifying .gitmodules, update the WW3 submodule:

git submodule update --init --recursive

4. Register ACCESS-OM3 in Spack for Development

Now, register the access-om3-nuopc source code in Spack:

cd ../
spack develop -p ./access-om3-nuopc/ access-om3-nuopc@git.0.4.0
spack develop access-om3@git.2025.01.0

5. Concretize and Install Dependencies

Force a fresh Spack concretization (dependency resolution) and install everything:

spack concretize -f --fresh

Run Spack Install

Execute the following command to build and install all dependencies:

spack install

:warning: Note: The installation process will take a long time, especially on the first run, as Spack compiles all required components from source, including MOM6, CICE6, and WW3.

Rebuilding ACCESS-OM3 After Modifying WW3

Once you have successfully installed ACCESS-OM3, you may need to rebuild it after making changes to WW3. Follow these steps to recompile and update your modified version:

1. Enable Spack

If you have opened a new terminal session or logged in again, first enable Spack:

cd /g/data/$PROJECT/$USER/spack/0.22
. spack-config/spack-enable.bash

2. Activate the Spack Environment

Reactivate your Spack development environment before making modifications:

spack env activate ww3-dev

3. Modify WW3 Code

Navigate to your ACCESS-OM3 source directory and make the necessary modifications to the WW3 component:

cd spack-dev/ww3-dev/access-om3-nuopc/WW3/WW3

Once done, save the changes and commit them (optional):

git add .
git commit -m "Relevant commit message that describes the changes made."

4. Rebuild ACCESS-OM3 with the Updated WW3

Run Spack install to recompile the model with your changes:

spack install

:warning: Note: This step will only recompile components that were modified (i.e., WW3) and not rebuild everything from scratch.

Run the Model with the Updated WW3 Executable

Once you have made changes to the WW3 codebase and rebuilt ACCESS-OM3, you will need to configure the model to use the updated executable. Here’s how to proceed:

1. Load the Latest Version of Payu

module use /g/data/vk83/prerelease/modules && module load payu

2. Clone the access-om3-wav-configs Repository

Clone the appropriate branch from the access-om3-wav-configs repository. For example:

For the dev-1deg_jra55do_ryf branch:

payu clone --branch dev-1deg_jra55do_ryf git@github.com:ACCESS-NRI/access-om3-wav-configs.git wav_config_ryf

For the dev-1deg_jra55do_iaf branch:

payu clone --branch dev-1deg_jra55do_iaf git@github.com:ACCESS-NRI/access-om3-wav-configs.git wav_config_iaf

3. Update the Configuration

Modify the config.yaml file to update the executable path and remove the default module:

  1. Find the exe: line and replace it with the locally built executable
  • The executable is usually located in:
/g/data/$PROJECT/$USER/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-***/bin/access-om3-MOM6-CICE6-WW3
  • Update config.yaml in the wav_config_* directory accordingly:
exe: /g/data/$PROJECT/$USER/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-***/bin/access-om3-MOM6-CICE6-WW3
  1. Remove the access-om3/2024.09.0 module
  • In the modules: section, remove the following line:
- access-om3/2024.09.0

Regenerating Input Files for WW3

When modifying the WW3 code, you need to update the executable and ensure that the wave model uses the correct grid and restart files. After building WW3 with Spack, the next step is to regenerate key input parameter files (mod_def.ww3) and restart files (restart.ww3) using the updated executable.

This process involves:

  1. Running ww3_grid to generate a new mod_def.ww3 (grid input parameters).
  2. Running ww3_strt to generate a new restart.ww3 (restart conditions).
  3. Updating config.yaml to use these new files.

Steps to Regenerate and Update WW3 Grid & Restart Files

1. Navigate to the WW3 Pre-Processing Directory

Move to the WW3_PreProc directory inside the wave configuration folder:

cd /g/data/$PROJECT/$USER/wav_config_*/WW3_PreProc

2. Generate the Grid Input Parameter File (mod_def.ww3)

Run the ww3_grid executable to create mod_def.ww3:

/g/data/$PROJECT/$USER/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-***/bin/ww3_grid

This will generate the mod_def.ww3 file in the WW3_PreProc directory.

3. Generate the Restart File (restart.ww3)

Run the ww3_strt executable to create restart.ww3:

/g/data/$PROJECT/$USER/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-***/bin/ww3_strt

This will generate the restart.ww3 file in the WW3_PreProc directory.

4. Update config.yaml to Use the New Files

Modify config.yaml so the model picks up the newly generated files. Locate the following lines:

- /g/data/vk83/configurations/inputs/access-om3/ww3/initial_conditions/global.1deg/2024.04.18/restart.ww3
- /g/data/vk83/configurations/inputs/access-om3/ww3/mod_def/global.1deg/2024.04.18/mod_def.ww3

And replace them with:

- /g/data/$PROJECT/$USER/wav_config_*/WW3_PreProc/restart.ww3
- /g/data/$PROJECT/$USER/wav_config_*/WW3_PreProc/mod_def.ww3

5. Verify and Proceed

  • Ensure mod_def.ww3 and restart.ww3 exist in WW3_PreProc.
  • Confirm config.yaml reflects the updated paths.
  • You can now proceed with running the model using:
payu sweep && payu run

You’ve successfully updated the WW3 executables and regenerated necessary input files for ACCESS-OM3. Ensure the correct executable and files are referenced in config.yaml before running the model with payu. This process keeps WW3 aligned with your modifications.

2 Likes