COSIMA TWG Meeting Minutes 2025

Date : 24 Sep 2025

Attendees: @sofarrell , @NoahDay , @dougiesquire , @ezhilsabareesh8 , @aekiss , @minghangli , @helen

Chair: @cbull
Minutes: @anton

CM3 Analysis - Chris

  • There is a CM3 analysis scoping meeting planned for in about 3 weeks. Check hive forum for the invite soon. Update: here’s relevant post.

OM3 Config Docs - Anton

  • We are going to move the all the documentation stubs back into main branch. The current stubs based system have incomplete versioning, are hard to edit and have weird quirks (like links breaking all the time)

  • Anton suggested CalVer versioning for the docs - when doing a config release, a docs release would be manually triggered at that time. (e.g. release-MC_25km_jra_ryf-1.0-beta would align with 2025.08.000 release)

  • Fixing docs is possible and increments the minor version only (e.g. 2025.08.001 is a future release, still related to release-MC_25km_jra_ryf-1.0-beta but with documentation only corrections)

  • In future, config and doc releases happen about the same time (e.g. release-MC_25km_jra_ryf-2.0-beta and 2026.0x.xxx docs)

Other business

  • Andrew Kiss thinks the bottom roughness calculation for OM3 inputs might be wrong. We need to change the length scale from one based on resolution to one based on the wavelength of the internal tides. AK will make an issue soon

  • Dougie - 100km branch sync. Dougie showed issue #771 and discussed doing a long-run soon, but EOS doesn’t impact this.

  • There is an OM2 build now with generic tracers based BGC. Should be able to use it with wombatlite, wombatmid, bling, cobalt etc. No config releases yet

  • Roster rotation happening next week

Date: 8 Oct 2025

Attendees: @sofarrell, @AndyHoggANU, @ezhilsabareesh8, @aekiss, @mmr0, @minghangli, @kieranricardo, @MartinDix, @helen
Chair: @cbull
Minutes: @dougiesquire

Sea-ice in OM3 runs

Kieran is curious whether we are seeing same issues in Northern Hemisphere as in CM3 (discussed in previous meeting I think)

No one has looked

Baltic sea salinity

Baltic sea is losing salt in CM3. Amount being lost is similar to the restoring applied in OM3.

Kieran: Shall we do some bulldozing?

Target is Baltic sea salinity stable around 8 or 9 g/kg

Ezhil will try deepening some of the particularly shallow cells

Ezhil also to update his OM3 salinity restoring comparison plots to compare to OM2 with “new” bathymetry

make_diag_table auxiliary variables

PR to add the option to include auxiliary fields with make_diag_table is here: Allow optional auxiliary_fields in each file by dougiesquire · Pull Request #14 · COSIMA/make_diag_table · GitHub

Upcoming releases?

25km RYF beta

Minghang is currently doing scaling tests to inform NCMAS proposals. Would be good to have a released version for this.

Agreed to do a minor release soon (1.1-beta)

100km alpha

Probably shouldn’t release without updating the grid

Won’t release anything soon

Panan alpha

No control run

Will target alpha release when this is in a configuration branch

NCMAS applications

Want to avoid multiple groups putting in applications that use ostensibly the same models and have wildly different compute costs.

Approximately 4 or 5 groups putting in NCMAS proposals.

Any information is needed within 10 days.

We’ll try to provide

  • Scaling info for:
    • MOM6 in Panan OM3 8km. Vary core counts for MOM only.
    • WW3 in OM3 100km. Vary core counts for WW3 only.
  • Cost for:
    • OM3 25km
    • OM3+WW3 100km
    • Panan OM3 8km

Still keen to hear from people about their plans for NCMAS. Trying to do the best we can with limited time

MOM baroclinic timestep

100km configs use MOM baroclinic timestep of 1800s and coupling/CICE timestep of 3600s. Increasing baroclinic timestep to 3600s causes MOM to crash. Should we spend time looking into increasing baroclinic timestep.

Decided that it was not a good use of time at this point.

Where to put observational products?

Where should one save observational datasets required for analysis notebooks (motivation is some sea-ice analysis to be done by Anton and friends + @aekiss has a MLD dataset that would be good to try and add).

Ping Josh on Zulip and ask: Log in | Zulip

Date: 22 Oct 2025
Attendees: @NoahDay, @sofarrell, @Paul.Gregory, @aekiss, @anton, @helen, @micael, @minghangli, @MartinDix, @ezhilsabareesh8
Chair: @cbull
Minutes: @dougiesquire

NCMAS applications

@micael gave overview of the software transformation team’s responsibilities. They will do scaling for models in conjunction with model dev teams. Will maintain repository of notebooks showing scaling studies.

Proposals require cost and scaling. Don’t need scaling for every configuration. In previous years applicants have shared around tables of the cost for each model. Usually this information exists from previous runs people have done.

Can we have a coordinated dissemination of the information to the community?

@micael will share draft of a Forum post on model scaling notebooks tomorrow. @minghangli and @ezhilsabareesh8 will review/give input.

Use normal queue for postprocessing

@dougiesquire will update the configs to use normal queue rather than express for collation and postscript jobs.

Workflow docs meeting for access hive

Chris is meeting @paigem tomorrow to talk access hive workflow docs (e.g. how to run payu, publish an experiment etc) get in touch if you have ideas/want to join.

Cm3 dev team minutes are moving to a more open format

See new announce page and @MartinDix will be posting minutes. ACCESS CM3 technical meeting minutes

Residual sea ice

There is an upstream PR to CICE to zap residual sea ice (non-physical regions of trapped low concentration ice). @anton has done test runs with this on in ACCESS-OM3 see here.

Investigation is still ongoing - see the above linked PR.

Run logs

We should be pushing run logs for experiments we run. Should we have a shared repo, a la ACCESS-ESM1.6?

@cbull will set up a repo in the ACCESS-Community org and will ping those that have run experiments

Compute allocations

@cbull showed what has been used in tm70 so far this quarter.

ACCESS-NRI is planning to ask an additional 5MSU for Q1,2-2026. Storage will remain the same.

CM3 drop in hackathon

Tomorrow at 1pm. Details here.

Updating OM3 initial conditions

@dougiesquire will update initial conditions to include potential temp, conservative temp, practical salt and absolute salt.

This makes it easier to set up the correct initial conditions and restoring for a given model configuration. Also, facilitates evaluation.

@dougiesquire will also look into reconciling the duplicate downloads of WOA23 in ik11 and av17.

New IAF run

@aekiss and @ezhilsabareesh8 probably need to do a few more test runs for bathymetry edits (Bosphorous,…). These will be prototype runs, just to check edits are having desired effect.

@aekiss possibly also worth looking at a few more diagnostics in the existing IAF run.

We want to have a release before doing another long run.

@cbull will open an issue for listing changes we want to make before an IAF release and others can contribute what they would like to see in the new run.

CM3 team will be interested in picking up the released version.

Date: November 5

Chair: @cbull

Minute taker: @helen

Attendees: @aekiss , @AndyHoggANU, @anton, @cbull, @Helen, @MartinDix, @minghangli, @sofarrell

CM3 dev evaluation meetings.

Next Tuesday there is an open CM3 dev evaluation meeting at 10:30am

This could be of interest to a broader COSIMA community. Action: @cbull to advertise this to COSIMA community

IAF beta release

@ezhilsabareesh8 is release coordinator for IAF 25k beta release. The purpose of this release is to give the experiment a human-readable name and makes it easier to align with other models such as CM3. There are plans to do a short test run before a release and after some bathymetry changes. There is a list of priorities on the project board (PO is priority, P1 is nice to have). Aiming to release in about 2 weeks.

Main changes for this release are an updated executable and bathymetry modification

OM3 evaluations

Hovmoller plots show funny temperatures in the vertical (at ~500 m). This is probably due to diffusivity from not parameterising the internal tides quite right. @aekiss and @AndyHoggANU to investigate further

The Indonesian throughflow has flows going through the wrong channels. Probably a bathymetry issue which was also present in OM2. @aekiss and @AndyHoggANU to formally raise this issue via the release project board. UPDATE: @aekiss created an issue.

@cbull has recently run all the OM3 evaluations on makefigs but half didn’t work. Most errors can be accounted for by a grid error that will be fixed in future runs. However, these evaluations will be needed over the next 2 weeks. @aekiss and @cbull to discuss next week.

Sulu sea

There is a channel in this region that is open but not by much. However, there is not a large resovear of water building up so happy to leave as is.

Discussion on potential future experiment runs

These are:

  1. There will probably be a CM3 100km run and an OM3 100k IAF could be run for comparison

  2. Sensitivity runs on mixing/subsurface temperature issues

Some of the planned future releases that may also need control experiment runs:

  1. 25 km config for upcoming release

  2. 8km config for future alpha release

  3. Panantarctic domain for future alpha release.

There is a table of current experiments here:

Date: 19 Nov 2025
Attendees: @AndyHoggANU, @anton, @aekiss, @cbull, @ezhilsabareesh8, @jbisits, @MartinDix, @NoahDay, @Paul.Gregory, @sofarrell
Chair: @cbull
Minutes: @dougiesquire

Wave-ice coupling update from Noah

Now have waves making it into the ice. There was an issue with how the coupling was configured - mismatch between the field coming in and what WW3 thought it should be.

Antarctic sea-ice extents are about 1/3 of what they should be (Arctic is less affected). Still working out why.

MOM6 numerical mixing diagnostic

Joey is adding a diagnostic for numerical mixing developed by @janzika into MOM6.

He has an offline version of the diagnostic and has added it online into MOM6. He is trying to run it in the OM3 25km IAF config.

He is noticing that some existing diagnostics that he uses in the calculation of his diagnostic (but shouldn’t modify) have changed after his code modifications. Currently tracking down.

25km IAF release

Discussed Project Board release section

Bathymetry edits are close. Still tweaking a few things in Indo straits.

  • Lombok needs to be narrowed - Already done by Ezhil.
  • Leti sill needs to let more flow into Timor - Already done by Ezhil, though possibly widened too much, maybe try just 1 or 2 cells.
  • Makassar needs to be opened up - Still to do.

Bosphorous needs restricting but is already one cell wide. Do we need a porous barrier? No one has the expertise on how to do this, perhaps it was done at a config we imported but has been turned off. This can probably wait until after the release.

Chlorophyll for SW penetration - Very nearly done by Dougie. Should be done today or tomorrow.

Documentation needs to be reviewed by the ocean team.

@cbull will coordinate config doc review focusing on the “old stub” that @minghangli wrote but never got fully reviewed.

@ezhilsabareesh8 and @aekiss will try to finalise the bathymetry by Friday. We’re aiming for a release early next week.

MOM6 dev meeting

TURBO

John Dennis (NCAR) presented an overview of the TURBO project (something like “Towards ultra high resolution CESM with MOM6 and ocean BGC”). The project is to allow MOM6 to run on GPUs. This is separate from the ACCESS-NRI/GFDL effort to port MOM6 to GPUs.

After some initial scoping of various technologies, have decided to use AMReX:

  • C++ productivity framework - framework for block structured AMR (LBNL)
  • Supports CPU and GPU
  • Assumes logically rectangular grid
  • Includes support for IO distributed memory - don’t need to write your own message passing code
  • Can mix Fortran and C++ quite easily. Can write individual subroutines in C++ or Fortran, adding OpenACC directives to fortran subroutines
  • Good CPU performance - comparable to optimized Fortran and C
  • Good GPU performance

Anticipating non-b4b changes:

  • Assess with Ensemble Consistency Test - GMD paper (2016) Evaluating Statistical Consistency in the Ocean Model Component
  • Want to create ECTs for MOM6. Call for help to develop these tests.

Loose timeline for TURBO:

  • Starting with FMS infra layer. Goal is not to provide an complete alternative to FMS, but to replace pieces at a time.
  • Within 6 months hoping to have unit tests for the FMS infra rewrite and a prototype of a more-extensive rewrite
  • Within 2 years hoping to complete both C++/AMReX-based and Fortran+directive-based implementations so that comparisons can be made.

Plan to collaboration with GFDL/ACCESS-NRI effort as thought partners for algo dev.

ACCESS-NRI/GFDL GPU port update

Dynamic core is running. Seeing encouraging performance numbers.

In particular, finding that loop orderings that are fast for CPU are not good for GPU and vice versa.

Double-gyre test case is now running really well on GPUs. Diagnostics are difficult (GPU→ CPU→disk)

COSIMA/Cryosphere workshop

There will be a combined COSIMA+Cryosphere WG workshop 16th-20th Nov in Hobart

Still working out the venue. Inputs on venues welcome

Sessions on understanding MOM6

There is a plan to start open sessions on understanding the MOM6 codebase, possibly as part of the COSIMA WG meeting. Stay tuned. Get in touch with chris/adele to be involved in the scoping meeting.

Compute Usage

Is anyone planning any large runs in the near future?

Andy would like to run GM/Redi testing on ACCESS-OM3, once the new release is out.

CM3 will also be doing a long run

Storing intermediate products from inputs generation

Some of our workflows for generating configuration inputs require generating intermediate products that can be useful more broadly (e.g. for evaluation). Currently these are being stored in an ad hoc way in ik11.

Agreed to create a /g/data/vk83/data directory as previously suggested by @Aidan.

CICE C-grid

Siobhan asked for an update.

Anton: hasn’t progressed that much. Still have same instability, probably related to ice strength or ridging. Could also be related to coupling - stand-alone case didn’t crash. Still on the todo list. Andy suggested it would be good to have in the full release of the model but how it’s completed is yet to be decided.

new iaf run after the release is out

@cbull suggested it would be best to do the new iaf run (after the release) from a cold start, this will make it more readily comparable with cm3 whom are planning to another run. @MartinDix and @aekiss confirmed.

TWG Meeting Minutes Draft

Date: 3 December 2025
Chair: @cbull
Attendees: @anton, @aekiss, @cbull, @ezhilsabareesh8, @jbisits, @MartinDix, @NoahDay, @Paul.Gregory, @sofarrell @helen @TommyGatti @Aidan @jbisits @dougiesquire
Minutes: @ezhilsabareesh8


1. Build Provenance & Transparency

  • Discussion: Improving visibility of build details for reproducibility.
    • Current logs are hard to parse; proposal to surface key info (e.g., Git refs, tags) in deployment comments with clickable links.
    • Potential addition of concretization graph and spec.lock details for reproducibility.
  • Key Points:
    • SPACK recipes and hashes already provide provenance.
    • Release database exists for official builds (link shared).
    • CLI tool idea: hash executable → lookup source URLs.
  • Action Items:
    • Tommy to draft issue for adding clickable component links in deployment comments.
    • Explore feasibility of CLI tool vs. web-based lookup.

2. MOM6 Standalone Testing

  • Context: Need for MOM6 ocean-only builds for node testing and idealized cases (e.g., ANUtub).
  • Options Discussed:
    • Modify OM3 Spack bundle recipe (won’t work).
    • Update MOM6 CMake to allow building MOM6 solo exe alongside MOM6 ACCESS3 library. Add Spack variant for MOM6 solo exe.
  • Consensus: Proceed with CMake modification and Spack variant approach.
  • Action Items:
    • Dougie to implement CMake changes and Spack variant.
    • Check resource impact (size, linking).

3. Release Update

  • Status:
    • Changes merged to release branch; reproducibility issues noted in metadata.eml.
    • Release notes finalized; DOI in progress.
    • Citation files (Zenodo.json, citation.cff) confirmed for Zenodo integration.
  • Next Steps:
    • Andrew to run stability test (~10 years) before official announcement.
    • Anton to monitor checksum/reprotest issues.
  • Target: Release announcement after successful stability test.

4. Output Directory & Notebook Updates

  • Issue: OL01 output path restructuring.
  • Decision: Move data and create symlink to maintain backward compatibility.
  • Action: Andrew to implement; update experiment table in config docs.

5. Evaluation Framework

  • Proposal: Project board for OM3 evaluation notebooks to ensure model-agnostic diagnostics.
  • Goal: Scripts that work across multiple models/resolutions.
  • Feedback: Positive; proceed with draft board.

6. Decoding Sessions

  • Plan: Short (10-min) sessions starting tomorrow.
    • First topic: GitHub code search (regex, limitations).
  • Notes: GitHub indexing caveats discussed; recommend disclaimer.

7. WaveWatch & Langmuir Mixing

  • Update (Ezhil):
    • Transition from KPP to EPBL in MOM6-I6 config.
    • Issues with wave coupling → increased mixed layer depth, reduced ice extent.
    • Current workaround: disable Langmuir mixing in MOM6 and WaveWatch.
  • Next Steps: Further testing; review papers; consult Albert.

8. Regional Configurations & Leave Planning

  • Helen to ensure key files moved from scratch space before leave.
  • Super Regional Australian model to be uploaded to GitHub.

9. MOM6 PR & Upstream Coordination

  • Plan to submit PR to MOM6 repo early next year; timeline depends on queue from other node centres.
  • Action: Dougie to prepare fresh branch; monitor MOM Dev discussions.

10. 100Km Wombat Run Strategy

  • Debate: Start with RYF vs. IAF; include Wombatlite?
  • Decision: Begin with RYF + Wombat Light, given compute availability.
  • Dougie to quantify cost and proceed; output to new folder.

11. Closing

  • Helen shared highlights from recent Forum for Operational Oceanography:
    • Interest in Pan-Antarctic model and CoastRI configurations.
    • Engagement with industry and regional modeling initiatives.

Date: 17 December 2025
Chair: @cbull
Minutes: @ezhilsabareesh8
Attendees: @cbull , @sofarrell, @minghangli, @anton, @dougiesquire, @aekiss, @taimoorsohail, @NoahDay, @angus-g

Agenda

  1. Updates on re-gridding issues across CMEFs and CDAPs
  2. Evaluation diagnostics discussion
  3. Wave mixing tests and Langmuir turbulence impacts
  4. IAF release status and review process
  5. OL01 storage space and quota concerns
  6. Layout changes and interaction with ongoing runs
  7. CM3 evaluation workflow and notebook updates

1. Re-gridding Issues Across CMEFs and CDAPs

  • Update from Minghang Li:
    • Meeting held with NOAA (Denise Worthen) regarding CDEPs and CMEPS remapping.
    • Current initialization times for high-resolution configurations are extremely long (hours), compared to NOAA’s setup (seconds to minutes).
    • NOAA uses matching meshes for forcing grids, minimizing remapping overhead.
    • ACCESS setup performs remapping twice (in CDEPs and CMEPS), causing inefficiency.
    • Masked vs. unmasked mesh mismatch leads to ESMF treating grids as different, rebuilding remapping handles unnecessarily.
    • Patch method for vector remapping is costly at high core counts.
    • Denise to test remapping in CDEPs.
    • Minghang to run components on different cores; results still slow.
    • Anton to investigate master mesh in data atmosphere.
    • Tests show updated CDEPs initialization reduced to ~2 minutes; CMAPS ~10 seconds.

2. Evaluation Diagnostics Discussion

  • Update from Taimoor Sohail:
    • Calculated cumulative heat and freshwater fluxes vs. global content changes.
    • Aim: Check global budget closure at each timestep.
    • Draft PR linked; scope unclear due to overlap with existing diagnostics.
    • Andrew clarified intra-model vs. inter-model conservation.
    • MOM6 has internal checks; ocean.stats.nc file includes global metrics.
    • Suggestion: Share notebook for review; split issue into sub-issues.
    • Dougie confirmed ocean.stats.nc includes useful global quantities.

Wave Mixing Tests and Langmuir Turbulence

  • Update from Noah Day:
    • Tested KPP vs. EPBL schemes (additive, rescale, none) using repeat-year forcing.
    • Findings:
      • EPBL additive → unrealistically deep mixed layers in Southern Ocean; severe sea ice extent reduction.
      • KPP and EPBL-none more reasonable; Arctic less sensitive.
      • SST biases confirm additive scheme problematic.
    • Options: Use KPP (different from OM3 standard) or EPBL-none (loses wave coupling benefits).
    • Siobhan questioned Langmuir circulation presence in KPP runs; suspects misconfiguration.
    • Suggest further checks on tropics and longer runs.
    • Chris suggested testing alternate initialization months.
    • Consensus: Investigate KPP Langmuir implementation and scaling impacts.

4. IAF Release Status

  • PR for IAF release is ready; Anton proposed tomorrow as review deadline.
  • Reviews mainly for scope approval, not technical detail.
  • GitHub release workflow discussed:
    • Merge triggers tag creation; manual update of release notes required.
    • Check Zenodo integration.
  • Dougie to extend wall time for collate job.
  • Agreement: Proceed with release as interim prototype.

5. OL01 Storage Space

  • Concern over quota due to large runs.
  • Wilton expected to remove restarts soon; no immediate action unless delayed.
  • Dougie syncing only 100km runs.

6. Layout Changes

  • Decision: Do not include changes in IAF release; apply immediately after.
  • No objections.

7. CM3 Evaluation Workflow

  • Ezhil working on making OM3/CM3 notebooks agnostic.
  • Need to merge Andrew’s pending PR to avoid rebasing conflicts.
  • Discussion on merging strategy; Git merge issues with JSON noted.
  • Plan: Attempt merge, resolve conflicts, proceed with flexible template design.
  • Volunteers requested for notebook conversion.

8. OM3 Model Release

  • Dougie raised question of minor OM3 release due to recent updates:
    • MOM6 ice-shelf bug fix, WaveWatch3 fix, MOM6 solo executable build.
  • Decision: Not urgent; defer until after upstream updates and CDEPS/CMEPS changes.
  • Minor releases considered beneficial for reproducibility.

9. EPBL vs KPP Mixing (Siobhan’s Update)

  • Compared diffusivities from EPBL vs. KPP:
    • EPBL provides better mixed layers, tighter thermoclines at high latitudes.
    • KPP smears mixing across column; less realistic in polar regions.
  • Recommendation:
    • Avoid major parameter changes; tuning limited to ±10%.
    • Further analysis needed for tropics and ENSO impacts.
    • Cross-compare with GFDL outputs.
    • EPBL approach preferred; continue investigation.

4 posts were merged into an existing topic: COSIMA TWG Meeting Minutes 2026