Minutes for 2025, please set new posts to ‘Wiki’ for easier editing.
TWG meeting - 15 Jan
Attendees: @helen, @anton, @cbull, @aekiss, @Minghangli, @KieranRicardo, @Martindix, @Angus-g, @PaulLeopardi
Landmask error:
Kieran has been running the CM3 prototype with the new 0.25 grids
Some points in Africa look like they should be land and are currently ocean
@aekiss will follow up, and mask out those cells.
Interesting that it only came up in CM3 and OM3 didn’t give an issue.
Kieren will try smoothing within the mediator for runoff.
Mom symmetric:
For mom6 regional , the recommendation is to use MOM “symmetric memory” but for access-om3 global modelling we haven’t been using this (NCAR is doing the same). In non-symmetric, all arrays are the same size but for symmetric the velocity arrays are one larger because they include all four edges for all cells (rather than just the typical north and east edges). The mom-ocean tests check that symmetric and non symmetric results are the same, and rotating domains give the same results. The halos for each “rank” are large enough that the missing cells (default 3 cells) for velocity won’t impact the results of any calculation.
Angus suggested that we could check the mom-ocean tests are testing the same parameters we are using for access-om3. In an overall sense, symmetric and non-symmetric should be bitwise identical.
@anton: will do a bfb test using mom_symmetric vs current for om3, with plan to turn on mom_symmetric.
@helen has raised an issue for the change
OM3-0.25 Project Board - check in on this. No updates/blockers from anyone
Run-off
Anton has some fixes to the OM3 configuration to conserve run-off. See Runoff not conserved · Issue #231 · COSIMA/access-om3 · GitHub
Will use first order conservative remapping for moving from JRA grid to the ocean grid. Any new grids need to go to atleast 80 degrees south to capture all runoff.
To account for differing landmasks, Anton suggested just mapping any water in land cells to the nearest ocean cell. We noted the limitations of this, which include runoff isn’t moved to the mouth of any bay that is landmasked, it is just moved into the nearest ocean. There is also no spreading of run-off, we need to test if EPBL + vertical river mix is stable and sensible with this configuration. We will go ahead with the simplistic just mapping to nearest ocean cell, and see how the results look.
Pedro at UNSW has been inserting frozen runoff at depth in OM2 iceberg spreading projects
Speaker for the Community Symposium on Sea Ice Modelling
ACCESS has been invited to present, Andrew could say something for the last seven years? or maybe someone from CSIRO has more historical perspective.
@anton to ask how long is the talk and follow up
GitHub Desktop and GitKraken - @cbull
GitHub Desktop
Provides a clicky interface to cherry-pick commits, reorder commits, squash, create branches, commits etc
Also shows local changes and other git stuff
GitKraken does similar + shows history graphs
MOM6 testing on the access-nri branch
The automated testing errors are hard to follow for folks who are not the authors. In future, we should check that tests pass as they are committed.
These tests are failing on 2025.01 branch can be resolved when @dougiesquire / @ezhilsabareesh8 is back. Further details of outstanding issues here.
ACCESS-OM3 0.4.0 Release
New components versions - updated MOM6/CIC6 and CMEPS versions.
Andrew & Chris to review (details)
MOM6
Chris is to start attending MOM6 developer moeetings, they are on fortnightly. And Dougie will resume attending as well.
Access-nri model transformation team will start working on a MOM6 GPU implementation. Led by @edoyango and @micael
@claireyung has put her work to track the MOM6 panAn/regional ice shelf model development (see issues for current status)
COSIMA training program
Starting soon, program is here:
Please let Chris know if there’s a particular session/topic you’d like to present on.
COSIMA talk upcoming (Thursday 13th Feb)
@minghangli present a brief introduction into the expt manager tool & save a more detailed tutorial at training program (23 May)
Need some content for the rest of the session (payu ?)
Andrew is Chair and Minghang in minutes for next TWG (29th Jan)
TWG meeting - 29 Jan
Attendees: @micael, @CharlesTurner, @helen, @anton, @cbull, @aekiss, @Minghangli, @Martindix, @dougiesquire
Agenda can be found in this link COSIMA TWG Announce - #55 by aekiss.
For the Cosima meeting:
Training program outpline:
@CharlesTurner has almost finished the cli for making intake datastores. He’s currently writing the tests. He’s also working on the coordniate search within intake ESM, allowing catalog to support searchable coordinates.
For the training program, it is good to have breakout rooms to allow people to retain knowledge better and allow informal discussion. There may also people jump in to ask other questions related to intake, it is better to leave some time for that as well.
@CharlesTurner is also working on the model autodetection through CLI when buidling a catalog. @dougiesquire raised concerns about validation risks, as model names are not strictly prescribed.
@MartinDix found differences on the tripolar edge between @aekiss and @ezhilsabareesh8 versions. @aekiss suggested waiting for @ezhilsabareesh8to return for further discussion.
3.2 diag_table
Discussions on diag_table daily / monthly output frequency and potential performance concerns when switching to daily output.
Ongoing discussions on managing diag_table and diag_table_source.yaml, where the yaml configuration file along with GitHub - COSIMA/make_diag_table: Python script to generate MOM diag_table is used to generate the diag_table. An issue is created Diagnostic table files management · Issue #259 · COSIMA/access-om3 · GitHub, where there will be further discussions there.
3.3 KPP and epbl with fixed runoff
@minghangli analysed salinity comparisons at the Amazon river mouth over time and across z* coordinates. Results show similar trends with KPP. The rivermixdepth parameter (40m or 20m) has minimal impact. Seasonal cycles are stronger, but salinity remains low (~0.02 PSU).
epbl with/without fixed runoff has velocity truncation errors, but KPP does not. Before the runoff fix, truncation errors appeared around Antarctica. Since runoff is a constant in time, errors may be circulation-related. @minghangli will provide a spatial plot of the new truncation errors with runoff fix. One thing to note, before runoff fix, truncations happened in a row of 10 years and seemed persistent.
@MartinDix mentioend that Kieran has been testing the horizontal spreading without epbl.
Next TWG is assigned: Chair: Dougie. Minutes: Andrew K. Andrew will give summary of this TWG at next COSIMA meeting.
ML: Using latest Riechl et al (2024) ePBL parameters but we get truncation errors. To maintain runtime performance comparable to OM2, we’ve set a tracer timestep of 3 hours, which is longer than GFDL’s 2-hour timestep. He’ll test the alpha release (0.4.0) since it includes fixes with more numerically stable schemes that might help address this issue.
CB: ePBL preferred over KPP in COSIMA meeting
CB: what are other groups using?
ML: GFDL are using ePBL with same params
ML: currently tuning params but still not working
Many items will be resolved with config updates
also on zulip
ML: OM3 build & config passed CI repro test but aren’t actually reproducing - CI test now fixed
but still have no repro between 0.3.0 and 0.4.0
also no repro in MOM6 standalone using MOM6 driver (not NUOPC, no sea ice)
default MOM6 parameters changed but have been fixed.
DS: only one of the long list of parameters is different, and this change won’t affect our model
ML: still no repro even with parameter issue fixed
DS: how did this get through the MOM6 repro testing?
DS: unclear whether we expect MOM6 repro given the version change - would have to check through all the
repro tests don’t even run on 0.4.0 config due to truncation
AK: would be good to have a way to tell if we expect repro between any 2 commits
MS: is run deterministic?
ML: yes
DS: do we want to dig into this to find out why it all changed
CB: would want to know whether we expect reproducibility. If repro not expected, do longer run to see if results are plausible. But gap between the 2 MOM6 version, so not a good use of our time to dig into all those commits.
DS: but how to know if repro is exected with digging into commits
CB: did Marshall mention this a month ago?
DS: might have been discussed at a MOM6 dev meeting we didn’t attend - ask Angus?
AS: ask Marshall
also go back to 0.3.0, add one PR and check whether it’s something in our process that is breaking things
CB: ok will ask Marshall
DS: also check ML’s standalone runs to see if they reach the same conclusions.
This problem will crop up with other components
AS: we could have CI run 1deg for 20yr on every update
MS: for every CI, make it fail to check it works
DS: there are many bugfixes in MOM6 that are turned off by default for repro but which we should turn on, breaking repro
CB: could any of the patches on patches involved in 0.3.0 → 0.4.0 be a problem?
DS: possibly, but unlikely, and not in
CB: notes on zulip
bug discovered (doesn’t affect us)
update to MARBL, will change cap, may affect WOMBAT - asked us to check
Marshall gave presentation on GPU work - see link to his notes. Impressively fast progress, eg pressure solve running on GPU. Targeting momentum solver first.
Our software team also contributing
Ed has long todo list
Some things in specified GPU coding style are unsupported by hardware; hard to get vendor support; NVIDIA won’t look at code due to license use (LGPL) - looking to move to more commercially-friendly Apache, which does not oblige disclosue of code changes (see table here). Asking ~90 contributors to approve license change.
DS: Generic tracer: code moving out of mom into ???, may affect us.
DS: Next set of changes will alter defaults - need to keep an eye on this for repro.
AK: good reason to storeMOM_parameter_doc.* in repo
DS: release team suggest doing via payu
MS: do via pre-commit hook to run model for 1 timestep?
AS: then add repro CI test to fail on diff between these files
MS: set up cron to regularly check?
DS: but want to know immediately that defaults have changed
MS: belt and braces - cron job to pick it up in case repro test was forgotten in commit
AS: on release there are more stringent tests than commit, so that would pick it up too
AK: would be nice to also be able to do this with CICE
DS: are we talking about just committing the docs (easy, we can do in payu) or CI repro test against branch to merge in (harder, involved release team)
CB: would be happy to have a go at this but AS is probably better positioned, AS will write something and CB can help review/have a chat.
CB: Dougie to give TWG update to Thurs COSIMA meeting
ML: increase tracer timestep may be part of problem with ePBL; reducing from 3 to 2hr (matching GFDL) fixes truncation errors, but performance is worse. Truncation occurs at particular places around Antarctica.
DS: truncations in 0.30 and 0.4.0?
ML: haven’t tested in 0.4
DS: one of the new MOM6 changes (off by default) helps improve model stability
ML: should I discuss this at CSIMA meeting tomorrw, or spend some time working on it first?
CB, AK: might be better to discuss ePBL in offline meeting with the few people who can give good input
ML: Wilton found ePBL performed similarly to KPP but without vertical resolution dependence
DS: mom dynamic timesetep is quote short (30 min) and also differs from coupling, unlike 0.25
DS: probably inherited from CESM
DS: CI to automatically create diffs (like the ones in README) between PR branches and all config branches. DS is coordinating. Some interest from Cable/land teams.
Invitation to present at Blue Link in late March on OM3 development and/or high-resolution development.
To be discussed offline
CB: due Friday. @minghangli please write an update on the OM2 new control experiments.
CB: starting next week on Fri 21st Feb (discussion link, draft program). What’s the status AS?
AS: may need to use xp65 env. Will have some slides. Possibly a notebook
@dougiesquire on for the agenda for next Wednesday’s OSIT and for the next TWG: Chair: Anton. Minutes: Dougie.