Ocean Team Workshop (12-15th November, 2024)

COSIMA TWG
,
1

ACCESS-NRI Ocean Team Workshop (12-15th November, 2024)

Location: ACCESS-NRI, executive board room. Canberra. (Backup: @AndyHoggANU’s office and Hales seminar room, RSES)

Purpose

  • Reflect, plan and coordinate the development work plan of the Ocean Team.
  • Progress topics that are best addressed in-person as a team, particularly tasks that benefit from inter-team and external interactions.
  • Make progress on issues we’ve been putting off!

Participants

Core: @cbull @minghangli @anton @ezhilsabareesh8 @aekiss
Optional: as tagged below but if you are interested in contributing to specific elements, get in touch. Remote participation is possible.

Topics

Smaller meetings:

  • Ezhilsabareesh Kannnadasan and Andrew Kiss on bathymetry for OM2-025.
  • CI demonstration

Program.

This is an approximate program and will change a little as needs and interests permit but is based on indicated availabilities. Sequencing is unintuitive as @AndyHoggANU is not available on Tuesday.

Tuesday 12th November

9.30am – 9.45am. Welcome!
9.45am - 10.15am. CSIRO - ACCESS-NRI standup
10.15am - 10.30am. ESM 1.6 release blockers? (following stand up)
10.30am - 10.45am. Morning tea.
10.45am - 11.30am. @anton CI demonstration.
11.30am - 12.30pm. CM3 coordination and development (invited: @spencerwong @kieranricardo @MartinDix @heidi )
12.30pm - 1.30pm. Lunch
1.30pm - 3pm. ACCESS-OM3 model evaluation paper metrics I. Presentation by @rbeucher. RE: Model Live Diagnostics vs ESMValTool etc. (@ClareCR?)
3pm - 3.30pm. Afternoon tea.
3.30 - 5pm. ACCESS-OM3 model evaluation paper metrics I continued

6pm. Social event: drinks at Badger and Co. All welcome! (Org: @Lauren)

Wednesday 13th November

(Boardroom is booked from 1.30 - 2.30, hence, later lunch.)

Following extra people invited: @MartinDix @AndyHoggANU @angus-g for this morning’s session.

9.00am – 9.30am. Current 2024-2025 workplan, introduction and revisions needed?
9.30am – 10.00am. Present results from user survey: [ACCESS-NRI ocean modelling survey: steering future development for users]( Survey link)
10.00am - 10.45am. Future plans and priorities.
10.45am - 11.30am. MOM6 testing strategy and related MOM6 node dev’ work. (Extra: @marshallward @micael)
11.30am - 12.00pm. Late morning tea (due to late lunch).
12pm – 1.30pm. Planning for spackifying OM3 build process I (extra people invited:
@Aidan, @harshula, @TommyGatti, @micael).
1.30pm - 2.30pm. Lunch!
2.30 - 4pm. Implementing the discussed spackifying OM3 build process II
4pm - 4.15pm. Afternoon tea.
4.15pm - 5pm. spackifying OM3 build cont.

Thursday 14th November

9:00am - 9.45am. Future plans and priorities. @aekiss @AndyHoggANU
9:45am – 10.15am. ACCESS-OM3 model evaluation paper metrics II (invited: @AndyHoggANU) and global sanity checks I (e.g. freshwater budget, energy conservation etc).
10.15am - 10.30am. Morning Tea
10.30 - 11.30am. More paper metrics II. / parallel offline chats
11:30am- 12.30pm. COSIMA meeting (Matt Harrison (GFDL) - Improved Surface Mass Balance Closure in Ocean Hindcast Simulations).
12.30 - 1.30pm. Lunch.
1.30 - 2pm. Further spackifying OM3 build process III? Do we need more input from @Aidan, @harshula, @TommyGatti?
2 - 3pm. Skillshare.
3pm - 3.30pm. Afternoon tea.
3.30 - 4pm. Writing up minutes/notes from past sessions.
4 - 5pm. More spack OM3 III.

Friday 15th November

NB: @dougiesquire is available.
9.00am - 9.15am . @NBateman-bit “team” photo and other comms materials (example of outreach opportunities).
9.15-11.00am. More spack OM3 IV.
11.00am - 11.30am. Morning tea.

(NB: 11:00am - 1pm - @cbull has CMIP7 project meeting)

1 Like
2

What’s the plan for the ACCESS-OM3 model evaluation paper? Will it be multiple resolutions like the Kiss et al. 2020 paper?

3

My thinking is likely ‘yes’ for the main configurations, and that the 2020 paper is a great starting point/template, however I think it depends on how development progresses and community input (see program, survey, here etc.). For example, it may make sense to have multiple papers if there are too many configurations to document in one paper or some configurations are best covered by a separate paper given their different focus/features (e.g. ice shelf cavities).

4

I agree - there could easily be several papers, each being a go-to citation for the major types of ACCESS-OM3 configs.

The starting point is MOM6-CICE6 - a description and comparison across multiple resolutions (a la Kiss et al 2020) would be a large paper in itself.

MOM6-CICE6-WW3 (impact of waves on mixing and floe size distribution) could be a separate follow-on reference paper.

A third could present the new WOMBAT.

Then there’s cavities, tides, vertical coords, etc.

3 Likes
6

Workshop slides:
2024.11.08_ACCESS-NRI OceanTeam Workshop (1).pptx (12.0 MB)

7

Tuesday 12th November: 10.45am - 11.30am. CI Demonstration​
People present: @Aidan , @TommyGatti , @helen , @aekiss , @cbull , @anton @ezhilsabareesh8 , @minghangli
Topics covered (briefly): ​Demo for source code CI and configuration CI, model deployment.
Next actions: ​
@anton:- Reference the model deployment template in the readme. Or add to the top of spack.yaml
Does it make sense to compare manifests as reproducibility.

To be followed up: ​
Summary note written by: @minghangli

Anton gave a quick demo of the source code build CI and the configuration CI using Spack.

CI does the spack build and doesnt need users to have their own spack installation.

1. Source code build CI:

spack.yaml contains the environment which defines the versions of individual source codes.

Under spec: specs lists packages that would be installed if the PR is merged. The digits represent a pre-release version @git.(year.month.version) rather than a git tag.

Under packages: update the git hash for the main dependencies, e.g., access-om3-nuopc from cosima access-om3, either a tag or a git commit full hash.

Down to the bottom, in projections, one needs to update the version of the overall deployment too. In this example, one needs to update access-om3 and access-om3-nuopc. Those numbers would be the release numbers. Some more documentations may be referred here. One example (i.e., a PR) is given below.

# configuration settings.
spack:
  specs:
    - access-om3@git.2024.09.0
    + access-om3@git.2024.10.0
  packages:
    # Main Dependencies
    access-om3-nuopc:
      require:
        - '@git.0.3.1'
        + '@git.1f36419a1f4ba2d0fd5136e4eb43d3b4a52a162c'

    # Other Dependencies
    esmf:
@@ -64,5 +64,5 @@ spack:
              'SPACK_{name}_ROOT': '{prefix}'
        projections:
          all: '{name}/{version}'
       -   access-om3: '{name}/2024.09.0'
       -   access-om3-nuopc: '{name}/0.3.1'
       +   access-om3: '{name}/2024.10.0'
       +   access-om3-nuopc: '{name}/1f36419a1f4ba2d0fd5136e4eb43d3b4a52a162c'

During the PR open and updates events, it creates pre-releases. However, if the PR is closed, whether it is closed or merged, it will clean up all pre-releases. And if it were merged successfully, it will go and attempt to do a proper release version. On the other hand, the closed or merged PR can still be recreated. For example, one can still use the spack.yaml to recreate the PR, the environment at a particular point. When the deployment is running, it will take ~1 hour to build all dependencies on GADI. But it will reuse anything that is not modified. For the timebeing, there is a bug related to parallel IO…

Chris asked if this deployment is expensive or not, Tommy replied: as much as you want.

The build is in parallel for each package using login node with 4 tasks, but the entire process is not. Now the parallel builds are suppressed due to memory errors, specific for the ESMF build.

2. Configuration build CI:

A channel config in MOM_input of 1 deg configuration was updated in the demo. One may refer to this PR.

To trigger the Bitwise Reproducibility, one needs to write down !test repro in a new comment. This will trigger a 3-hour run of the model and compare answers to the ones that are saved in the repo.

The reproducibility check compares to a saved checksum in the configuration repository. If a PR changes the checksum (and is expected to), use !test repro commit to update the saved checksum (and the bot will commit the new checksum).

Post-note:

We can save the MOM_docs output using payu, rather than CI:

8

Tuesday 12th November: 11.30am - 12.30pm. CM3 coordination and development​

People present: @helen , @aekiss , @minghangli , @ezhilsabareesh8 , @anton , @cbull , @heidi , @spencerwong , @kieranricardo ,@siobhan, and @MartinDix
​
Topics covered (briefly): CM3 development, OM3 topography, OM3 build, melt pond, salinity restoring and runoffs.​

Next actions:
Invitations for CM3 development meeting to more attendees @MartinDix
Share new OM3 topography for CM3 development @ezhilsabareesh8
Test epbl mixing @minghangli
Confirm best scheme (epbl/kpp) for wave-mixing @ezhilsabareesh8
Demo from @aekiss and @ezhilsabareesh8 on the topo editing tool

To be followed up: Melt pond work follow up by Kieran, Siobhan and Spencer. ​

Summary note written by: Ezhilsabareesh Kannadasan

Kieran Updates:

OM3 grid and topography updates (Ezhil):

  • OM3 topography is updated fro C-grid and new topography generation workflow is developed for C-grid.
  • Hand edits have been made to open the boshporous strait to include Black sea.
  • Salinity restoring is not improved in OM3 compared to the previous topography (OM2 Topo). It maybe related to the runoff issue in OM3 https://github.com/COSIMA/access-om3/issues/231 ?
  • Andrew: Use OM2 as guidance to edit OM3 topography but not necessarily all the edits from OM2 are required.
  • Kieran: A new 1 degree OM3 topography required?. Andrew: It may require more hand edits than the quarter degree topography, not a high priority
  • Martin: Is there a provenance of files for the newly generated topography? , Ezhil: Separate repo for the topography generation workflow https://github.com/ACCESS-NRI/make_OM3_025deg_topo
  • Test ePBL scheme in MOM6 to see it fixes runoff issues.
  • MOM6-CICE6-WW3 may require ePBL since Langmuir multiplier is passed from waves to ocean and the Langmuir mixing parameters are defined in WW3 inputs. UFS ocean-wave coupled model uses ePBL in MOM6 https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/parm/MOM_input_025.IN
9

Tuesday 12th November: 1.30pm - 3pm. ACCESS-OM3 model evaluation paper metrics I. RE: Model Live Diagnostics vs ESMValTool etc.
People present: @helen, @aekiss, @minghangli, @ezhilsabareesh8, @anton, @cbull, @rbeucher, @CharlesTurner ​
Topics covered (briefly): ​ Model Live Diagnostics tool available for monitoring simulations as they run and ESMvaltool for evaluating/comparing model output to observations/other models. Datasets and metrics that we should start with to evaluate ACCESS-OM3.
Next actions:
​ example run of om3 (@ezhilsabareesh8)
observational datasets (@cbull). Could ask people such as @RyanHolmes, @Pearse, @Matthis_Auger for input on most appropriate / updated datasets ()
example evaluation notebook of om2 that would need to be ESMValCore-ised (@anton)
template for ACCESS-OM3 evaluation metrics (@anton)
Summary note written by: @cbull

Summary @rbeucher talk of available tools and support offered by MED team.

@rbeucher presentations slides are here:
OceanTeamWorkshopMED.pptx (6.9 MB)

Model evaluation and diagnostics

Two major use cases:

  1. Model development (ensure model behaviour is sensible)
  2. Data user perspective (has been most of the team’s focus)

Validation (basic checks against observations) vs Evaluation (in depth comparison using research specific metrics). Evaluation is complicated! Intersection of theory, observations and models. Challenges in model evaluation: data format capability, data accessibility and documentation, different use cases.

Supported tools
MED Conda environments, ESMValTool-Workflow, ILAMB-Workflow, ACCESS MED Diagnostics, ACCESS-NRI Intake Catalogue, Data Replicas

Multi-Stage evaluation strategy involves 3 stages: initial diagnostics (interactive checks as the model runs, ideally uses Model Live Diagnostics), intermediate evaluation, detailed diagnostics

Re: Model Live Diagnostics. Python based tool that can be imported from a ARE Jupyter session. Just need to specify model type (OM3 is not currently supported but that could be added) and output folder. Can also compare multiple simulations as long as they are the same type and available in the access-nri-intake-catalogue.

Re: Intermediate evaluation. Climate aspects over longer simulation periods in which comparisons are made with previous model configurations, CMIP6 etc. E.g. COSIMA recipes.

Re: Detailed diagnostics. Examine specific processes with dedicated diagnostics. Idea: in-depth analysis of specific climate processes. Suggested tool is ESMValTool.

Re: ESMValTool-Workflow. Likely overkill for model development but useful for analysis. Pro: large community with many applications. Cons: steep learning curve (CLI based)

Re: CMIP fast track evaluation strategy. Historically, ESMValTool did not have a lot of ocean diagnostics. Work is ongoing porting COSIMA ocean receipts to ESMValTool. ESMValCore Python API is used for pre-processing. Aspiration is to add recipes for ENSO and sea ice. Showed example of the ENSO evaluation recipes that have already been created.

Re: IOMB-Workflow – International Ocean Model Benchmarking (IOMB) package. Evaluates marine bgc models against observations.

Replica datasets, considerable number of datasets have been processed by the MED team to make comparisons easier. Currently datasets such as WOA are part of the collection and additional variables could be added, as could newer versions. Romain had imagined having a specific sub-collection for COSIMA data. Ship like sections have been used in OM2 in the past. Have also developed a tool for on the fly CMORisation.

Can help with regridding new datasets: so we can send them a wishlist. For example: NOAA passive microwave data (sea ice concentration, Antarctic/Arctic sea ice thickness)

Model Live Diagnostics tool. Can do live plots and spatial plots. Users can add their own diagnostics which once accepted become part of the tool for anyone.

ENSO recipes is a proof of concept example that they are developing that uses ESMValCore (current assumption is that the data is in the catalogue), see:
GitHub - ACCESS-NRI/ACCESS-ENSO-recipes: Recipes and metrics for evaluating ENSO in ACCESS, which is looking to create figures like https://clivar.org/news/clivar-2020-enso-metrics-package

ACCESS-OM3 evaluation datasets we could ask the MED team to turn into replica datasets.

There is a list and discussion of metrics at ACCESS-OM3 evaluation​, the following is based on that.

ERSST v4 (SST comparisons)

CNES-CLS13 and AVISO SSALTO/DUACS (mean and variability of SSH comparison). Updated product? Perhaps this: https://www.aviso.altimetry.fr/en/data/products/sea-surface-height-products/global/gridded-sea-level-heights-and-derived-variables.html
i.e. The delayed-time “allsat”

WOA2023 (T and S comparisons). Already on Gadi (/g/data/ik11/observations/woa23)

NSIDC CDR v4 (sea ice concentration). NOAA/NSIDC Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 | National Snow and Ice Data Center

NSIDC sea ice index v3 (area and extent). Sea Ice Index, Version 3 | National Snow and Ice Data Center

Tropical Atmosphere Ocean data. Want: subsurface temperature, fixed depth currents, subsurface salinity. data. Related: Forum post

chlorophyll
Sauzede et al., 2016 JGR Oceans

Thursday 14th November​

9:45am – 10.15am. ACCESS-OM3 model evaluation paper metrics II (invited: @AndyHoggANU) and global sanity checks I (e.g. freshwater budget)
People present: @helen, @aekiss, @minghangli, @ezhilsabareesh8, @anton, @cbull, @AndyHoggANU ​

Existing GitHub repository that needs to be updated.

Also see figure scripts from OM2:

10

Wednesday 13th November: 9.00am – 9.30am. Current 2024-2025 workplan, introduction and revisions needed?​
People present: Helen, Andrew, Ezhil, Minghang, Angus, Anton, Chris, Martin, Marshall, Andy​
Topics covered (briefly): ACCESS-NRI workplan​
Next actions: ​
@cbull to capture status of workplan as needed for reporting
To be followed up: ​
Slides: CBull​
Summary note written by: Anton Steketee ​

See slides for workplan and status of items

Marshall asked if outcomes are dependent on collaboration with GFDL or others. Response is we would / will collaborate on these but it’s not essential for timelines.​

We also talked about 1/12th model and support for model analysis.

11

Wednesday 13th November: 9.30am – 10.00am. Present results from user survey: [ACCESS-NRI ocean modelling survey: steering future development for users]( Survey link)​
People present: ​
Topics covered (briefly): ​
Next actions: ​
To be followed up: ​
Summary note written by: Anton Steketee

See slide deck for survey results

12

Wednesday 13th November: 10.45am - 11.30am. MOM6 testing strategy and related MOM6 node dev’ work.​
People present: @marshallward , @MartinDix , @Aidan , @AndyHoggANU , @TommyGatti , @harshula , @angus-g , @helen , @aekiss , @micael , @cbull , @ezhilsabareesh8 , @anton , @minghangli ​
Topics covered (briefly): MOM6 testing framework, contributing rules, GFDL PR life cycle, regression tests, one example for performance test (FMS) (Fix for MASK_OUTSIDE_OBCS with MASKING_DEPTH · NOAA-GFDL/MOM6@4add0e4 · GitHub)​
Summary note written by: @minghangli ​
Replicate the verification tests and regression tests in dev/access?
Could ask the release team for input

Marshall gave a talk about the GFDL’s way to maintain the code, do their testings to change the code.
Link to the slides: https://www.marshallward.org/access2024/

image
image1676×1234 101 KB

  • First introduced the MOM6 Consortium, including a public version of MOM6 and development versions. The consortium is informal, unlike CICE Consortium which has a steering committee and other formal structures.

    • A simple example of the development process: changes are submitted to dev/gfdl, where they are tested and approved. Over some time, that branch will be synced with main.

  • Contribution Rules

    • Code must not change existing solutions
    • History must be preserved
      • Only merge commits between main and dev/*
    • Regression testing requires reproducibility
    • Forecasters require reproducible output
    • Researchers want latest features without answer changes

  • GFDL PR life cycle

    • submit to dev/gfdl for review
    • automated verification testing
      • anyone can do on any machine
      • git clone MOM6 then locate in .testing directory then run make -j test
    • code review
    • test for regressions on production machine
      • CI
      • Code style check
      • compilers
      • TC tests: are run on a very small domain size and are primarily used to exercise the code. They are not meant to generate realistic results. These tests are used as initial checks before code review.
    • rebase into node repo

  • Test outputs

    • ocean.stats
    • chksum_diag: “all” the diagnostics are enabled for bitwise reproducibility purpose.

  • Regression tests

    • tests with production runs on production machine using production compilers.
    • 61 tests in total, including ocean only, one way coupled to ice, and two way coupled ocean-ice.
    • tests are a bit longer, 100~ish timesteps

  • Github merge

    • dev/gfdl rebases everything to build a linear history of the model. But this approach may confuse users/developers when trying to understand the branch histories.
    • To preserve the history, all changes are merged into main.

There are performance regression tests that compare FMS timing clocks against the dev/gfdl branch as a reference. These tests were introduced after observing ~ 10% performance slowdown due to the addition of various diagnostics. But since these tests run on GitHub virtual machines, the results can be inconsistent and unreliable, making them less meaningful for reporting purposes. Hence although GFDL has these tests in place, they are not actively used.

@cbull asked about the procedure for approving pull requests (PRs) that change answers. The response from Marshall was that the goal is to implement changes in a modular way, ensuring that existing answers remain unchanged.

If a request is rejected and cannot be merged into the main, Marshall pointed out that this practice is not ideal. He provided an example where GFDL made a fix related to tracer vectors, but another group could not replicate the changes. Hence GFDL could not merge the fix until the answers were restored.

For the tests, they will run every time one does a push to the remote. GFDL also has additional tests that only runs for pull requests. But these tests are not automatic but they are manually chosen by each node.

What is the procedure when updating compilers?
GFDL has reference answers. But when things change beyond, GFDL will update their reference answers. For example, after switching to a new computing system with updated libraries and compilers, they obtained new answers and updated their reference answers, which had been in place for ~10 years.

All existing OM3 configurations support the reproducibility test by comparing the checksums within the repo themselves (restart reproducibility → 2 one day tests = 1 two day test).

The reproducibility tests for ACCESS-OM3 are tied to the model configuration repos. Hence, we currently do not have automated testing in place when the model build or configurations are updated.

13

Thursday 14th November: 9.00am - 9.45am. Future plans and priorities.​
People present: ​
@aekiss @AndyHoggANU @ezhilsabareesh8 @minghangli @cbull @MartinDix
Topics covered (briefly): ​
Next actions: ​
@cbull will summarise 0.25 deg om3 status on project board on github
@cbull follow up about anonymous help requests on access-hive
Semi-regular presentations from ACCESS NRI staff (e.g. intake, running a model, model config structure (information flow diagram – what levers affect what components - eg what files are read by payu/NUOPC/MOM6/CICE6/WW3/DATM/DROF), creating a spack develop environment, MOM6/CICE/WW3 code tours etc)​ (@cbull)
Summary note written by: Anton Steketee

  • survey - training time is planned for post-release of a new model. This might include. In person training at the nodes
  • follow up about anonymous help requests on access-hive - @cbull
  • @aekiss suggested a payu diagram - for input files & config files
  • can there be a zenodo citation for the cosima recipes, can it be easy for publications to share their scripts back ?

Priorities:​

  • 0.1 global is next. Is 0.1 WOMBAT then also for ‘free’? Yes! (on a technical level; parameters and spinup are another story)​
  • 1/24 global? ​
  • Features as opportunity arises.​

¼ parallel work ():​

  • Parameter tests (forum discussion)​
  • Updated build (component update)​
  • Run-off spreading ​
  • GitHub project board OM3 025? (Chris)​
  • Salinity restoring
  • 025 topography edits
  • Scaling performance
  • CICE c-grid

see slide deck for more notes

14

creating a spack recipe for model components tutorial

#enable spack
cd /g/data/$PROJECT/$USER/spack/0.22
module purge
. spack-config/spack-enable.bash

git clone ` https://github.com/ACCESS-NRI/ACCESS-OM3.git` 
spack env create test ACCESS-OM3/spack.yaml
spack env activate test
spack install --only dependencies #note that this will take a long time. #may need to add --fresh

setting up cmake
Alot of cmake is top level and the same for all components so we want to copy across

  1. Grab CMakeLists.txt from cosima/access-om3 and copy to CICE - make a cmake directory (see root_cmakelists_dir below*)*. (Done changes on new branch on CICE called “5-port_*cmake_*build”, where 5 is the current issue/pr number.)
  2. fix up header so consistent with CICE & add compiler flags from main (toplevel) cmakelists. Fix paths via local cmake variable (e.g. use {CMAKE_*SOURCE_*DIR})
  3. make issue in spack-packages repository for creating spack recipe. Use number to make a branch using that number “n-cice_spr”. Make spack recipe (spr) to link up the dependencies (see homework for instructions on how to do this)
    spack create —name cice https://github.com/access-nri/cice
  4. edit, package.py file for cice, to add git url & copy selected / appropriate depdendices from access-om3-nuopc package to new package file,
    add this line to include the cmake_lists from a subdirectory (presuming it was added to a subdirectory ‘cmake’):
    root_cmakelists_dir = "cmake" (ref CMake — Spack 0.24.0.dev0 documentation)
    4a) create cmake_args function to the package class to set cmake variables (e.g. PIO)
    COSIMA/ACCCES-OM3/cmake/FortranLib.cmake - the file itself needs to be copied to our (CICE) cmake folder.
    spack -d install cice@git.#hash
    (-d will give you more info)
    NB: we hit this issue whereby the local repo didn’t update with one of Aidan’s new commits. Another nb: Minghang had a similar issue with the spack.yaml in the spack environment (change both spac spec).

Here is Anton’s workaround:
spack env create cice-spack-package
spack env activate cice-spack-package -p
spack add cice@git.HASH
spack install cice@git.HASH
spack develop cice@git.HASH
go to the cice folder & git fetch latest
spack install

15

Presentation by @cbull 28/11/2024
Slides for COSIMA update on user survey and relevant workshop outcomes.
2024.11.08_ACCESS-NRI UserSurvey.pptx (6.2 MB)