CSIRO - ACCESS-NRI standup minutes

CMIP7 CMIP7 development
41

Meeting minutes (24/06/2025)

@clairecarouge
@inh599
@Jhan
@Aidan
@anton
@dougiesquire
@ezhilsabareesh8
@manodeep
@micael
@paulleopardi
@RachelLaw
@spencerwong
@cbull
Chair: @clairecarouge
Minutes: @cbull

MOM5 and cmake

  • Updating to MOM5 build with CMake (some context here). There is also updating the grid.
  • Rachel: happy for changes to go in given we don’t expect any radical answer changes.
  • Aidan (speaking for Harshula): does this affect generic tracers? Dougie: Currently, we are building from the development branch that Harshula has been looking into.
  • Ian: are there any downstream impacts on the model? Land sea-mask changes? Dougie: Land sea-mask has not changed. Re-gridding weights have been changed a little.
  • Claire: also supports bundling them together.
  • Anton: was hoping to do the build and CICE5 changes at the same time. Are there other build changes planned for UM / Cable? Ian/Rachel: Yes, they are coming but they are a while away.
  • Rachel: is it worth doing a 4xCO2 run in CICE5? Anton: is there a configuration already running with CICE4? Rachel noted that there is a config on GitHub but it doesn’t appear to be up to date with the spin-up configuration so Tilo had taken the dev-preindustrial+concentrations config and made the one line change required (changing CO2MMR value in line 652 in atmosphere/namelists). There is also some documentation in this forum post
  • Ian: slight backtrack. Would be good to separate technical and science changes.
  • @paulleopardi I missed your comment please add.
  • @dougiesquire suggested some changes are ready now but others such as the grid change (relevant to the med’ salinity issue) are blocked, would be at least a week for that one

Updates on oneAPI compiler switch

See details in a table at the top comment in this PR

With the new compiler the atmosphere is now “faster” than the ocean (was the opposite using the old compiler), this has led to a different load balancing with the new compiler. (Micael clarified that MOM5 is actually just slower so the atmosphere is unchanged.) Manodeep is going to look into whether giving the ocean more cores will improve performance, this will break bitwise repro’ so he’s been delaying this test. Manodeep wanted to know if it was possible that we use the old compiler for spin up and then when we go to production with the new compiler. Yes, that is a possible solution. @RachelLaw (@inh599 agreed) suggested having ~100 years of the new one before going to production. @micael wished to emphasise the the old compiler will stop getting improvements but in theory the new compiler should be capable of the same performance and so will hopefully “get it” in the future (@cbull and @micael note that this change down the line could be answer changing.)

Decisions Made

  • Proceed with infrastructure changes as they become ready, rather than waiting to bundle them. Separate technical and science updates where possible.
  • Keep the spin-up configuration updated and documented on the forum.
  • Avoid bundling too many changes at once to simplify debugging.
  • Maintain flexibility in compiler use during spin-up, but aim to finalize before production.
  • Accept that some answer changes may be inevitable with future compiler/machine updates.

Action Items

  • Dougie to proceed with MOM5 and CMake updates when ready.
  • Keep the spin-up configuration table updated on the forum.
  • Rachel (?) to locate and share the forum post documenting the 4xCO₂ runs. - this is already linked in the comment above. I’ve added some extra detail around the config status.
  • Manodeep to continue performance testing with oneAPI and explore optimal configurations.
42

Meeting minutes (01/07/2025)

Participants:
@anton
@manodeep
@clairecarouge
@paulleopardi
@MartinDix
@inh599
@pearseb
@Aidan
@adfraser
@dhb599
@dougiesquire
@harshula
@manodeep
@RachelLaw
@tiloz
@cbull

Chair: @clairecarouge
Minutes: @cbull

Action items from last week.

Rachel last week suggesting running 4xCO2 CICE5. Anton has done a run and will make some plots and put them on GitHub.

Next steps: Tilo configuration still needs to be finalised. Martin: encountered some Payu issues that are being worked through. Once they are sorted he will share the configuration [@MartinDix there were a few configurations here I think?]

Manodeep clarified: so everything else except the 4xCO2 is using CICE4. Anton: everything (all configurations) will be rolled over to CICE5 once we are happy with using CICE5.

@Paul: FYI will need details of configurations running and planned, to update the CMIP7 ACCESS-ESM1.6 ancillary suite and Python scripts. See https://code.metoffice.gov.uk/trac/roses-u/ticket/648 and Produce ancillaries for ACCESS-ESM1.6 submission to CMIP7 · Issue #23 · ACCESS-NRI/CMIP7-Input · GitHub

@spencerwong has been looking into freshwater spreading options. Looks like there’s some curiosities with conservation. Spencer is currently putting together what he’s found in an issue. @inh599: has looked into this in the past and is open to discussing further.

@inh599 update: have been working on Australian PFTs.

@manodeep update on the oneapi transition. Recap oneapi performance was 20-25% worse last week. Has tried adding a node and that leads to the same performance as before (with a higher SU cost).

@RachelLaw: @tiloz has flagged that there is 0 allocation on p66 for compute. We may need to ask other people for some resources.

43

Meeting minutes (08/07/2025)

Participants:
@cbull
@anton
@Aidan
@MartinDix
@pearseb
@alexnorton
@clairecarouge
@dhb599
@dougiesquire
@harshula
@inh599
@jemmajeffree
@manodeep
@paulleopardi
@RachelLaw
@rbeucher
@spencerwong
@tiloz
Chair: @clairecarouge
Minutes: @cbull

Manodeep optimisation update
oneapi with a new version is now matching classic compiler performance
19.6 vs 19.1 years per wall day. See PR for details

Manodeep is also looking into optimisation in terms of layout changes but this will break reproducibility. Martin: would entertain it if the performance was worthwhile so would be good to profile (as long as it doesn’t break determinism).

Aidan on metadata standards. Clare, Kelsey, Aidan, Martin, Romain have been working on an output specification document that describes all the ESM1.6 output. Aidan has been leading the global attributes spec’ where the plan is for this information to be injected into the file outputs by Payu using Axiom (e.g. model, model version, uuid etc). Aidan can share the doc via Sharepoint (needs an invite). Jemma suggestion: “I think it’s valuable to keep stuff that’s in the filename also in the metadata, just because it’s harder to accidentally rename the metadata.”

Claire: thinking behind netcdf version, is it for compression? Aidan: No there’s a few things, Netcdf 4 classic has a smaller subset of Netcdf 4 and is backwards compatible – however, compression is worthwhile. Anton: can turn on compression for cice (not sure of effort or timeline) but it’s part of a wider plan.

Ian: land (cable + mosis) via the UM makes things a little complicated so needs to be described more carefully (can take the discussion offline, Dougie is interested too). Rachel: is there any synergy here between cmor and the existing outputs? Romain: some but this process helps facilitate cmor-isation.

Spencer and Martin have found an old bug in the inland basin code for river (runoff?) which they are investigating.

Wouldn’t it be simpler just to assign salinity from the surface to every depth level in the Mediterranean? and run a control & new config from there?

Mediterranean sea (this discussion was also continued after the stand up with @MartinDix, @RachelLaw, @pearseb, @inh599, @cbull, @dougiesquire)
@ezhilsabareesh8 is leaving ACCESS-NRI so progress on this has been disrupted, see Ezhil’s summary. @cbull has been looking into it and was interested in what @inh599 and others had already looked at (sea surface salinity mostly). Rachel suggested ping’ing @matthew.chamberlain on the GH issue for the latest set of plots he’s made. The discussion converged on trialling bigger bathymetry changes (or aphysical mixing as @dhb599 proposed) and re-setting the whole med’ sea salinity back to initial conditions. Concern that the excess salinity in the western salinity may flow out into the Atlantic and affect the AMOC (we should try and avoid this) although a signature of western med salinity is visible in the Atlantic in the i.c. files.

Both @clairecarouge and @cbull can’t attend next week. Martin and Spencer can run next week (chair/minutes resp’)

44

Meeting minutes (15/07/2025)

Participants:
@MartinDix
@Aidan
@anton
@spencerwong
@manodeep
@paulleopardi
@RachelLaw
@rbeucher
@tiloz
@alexnorton
@jemmajeffree
@dougiesquire
@dhb599
@inh599
@harshula

CICE5:

@anton has run ~130 years of a 4xCO2 run with a CICE5 config. Results for SAT and TOA radiation are nearly identical compared to the AprilSpinup 4XCO2 run. Climate sensitivity is ~3.357 K for the AprilSpinup configuration and 3.290 K for the CICE5 run, and the difference is possibly within sampling noise.

Unclear on when to make the swap to CICE5. Masking changes will require a change to add_lprec, however potential updates to the UM will change this again. Should the swap be done at the same time as the UM changes and the change to oneAPI?

@Aidan in favour of swapping to CICE5 sooner rather than later, as it reduces the problem space.

OneAPI

@manodeep: The oneAPI has reached performance parity with the classic compilers. Results are not bit for bit identical, which is to be expected. We are now ready to do long runs to evaluate the science impacts of the change.

Cable: Australian PFTs

@inh599 has met with the Hawkesbury Institute at UWS who will be helping with parameter values. Some work is needed on this.

On the code side, @Jhan has found that runs which shuffle the indexes and should be bitwise identical are not. This seems to trigger differences in random places such as over the ocean, even though the code changes should only effect land.

@MartinDix to follow up with @Jhan, and suggests a memory overwrite could be happening somewhere.

Ambition is to finalise the Australian PFT work before the proper control run.

@RachelLaw: Aside from the Australian PFTs, we also want to change the vegetation distribution. @alexnorton is working on this. Additionally, there are problems in the CO2 seasonality especially at high latitudes, suggesting problems with leaf respiration that need to be chased down.

Cable: Library

@inh599 notes that work on cable as a library is still ongoing. @lachlanswhyborn is investigating bit reproducibility problems. Simultaneous developments on Cable main make it complicated to coordinate this work.

1pct CO2 configuration

A 1pct CO2 configuration is now available here, which @tiloz is testing. Temperature and CO2 are going up as expected.

It would be useful to add a convenient way of checking the gas concentrations that the model actually sees. Currently it’s difficult to confirm whether config changes are being picked up by the model.

Ancillary generation

@paulleopardi will set up a new branch on the ESM1.6 configs repository to test out generating forcings for the Historical configuration.

CMIP7 data request

@rbeucher has read through the CMIP7 data request and produced a spreadsheet of the requested variables, and their current status in ESM1.6. Please look over the spreadsheet (linked here )and add comments.

Fast track timeline

@Tilo notes that the fast track deadline is end of 2026. As we have a relatively fast model, and various configurations in runnable states, that allows us some more time to finalise the code. However we don’t want to drag it out for too long.

NetCDF conversion failures

netCDF conversion failures have been encountered by everyone running the model. @jo-basevi is investigating the causes and will submit an nci help request. @spencerwong has started a forum topic here to collect reports of the failures. Please add a comment with details of any failures you encounter. Reports close to the time of the crash would be especially useful.

45

Meeting minutes (22/07/2025)

Participants:
@anton
@lachlanswhyborn
@inh599
@MartinDix
@Jhan
@pearseb
@Aidan
@adfraser
@dhb599
@harshula
@jemmajeffree
@manodeep
@paulleopardi
@RachelLaw
@rbeucher
@tiloz
@cbull
@clairecarouge

Chair: @clairecarouge
Minutes: @cbull

payu dev bug

Spencer added things to calendar handling for the UM in esm1.6 late last week in Payu. This unfortunately broke running new experiments with Payu (existing submitted runs should have been okay), but was fixed yesterday and new payu dev should be working again.

water conservation in the um

Ian, Spencer and Martin have been looking into water conservation issue in the UM. Unravelling compensating errors. Could be a substantive bit of work to fix all the bugs. Considering rolling back to an earlier version. Jhan asked for further detail. CICE4 is doing a better job (Martin it’s not actually a CICE issue but the issue was found through the model change). Martin: Atmosphere is creating water, which could be balanced with scaling the river outflow (by a few %). Ian had a look at this and thinks it goes through about 14 layers of code. Martin: Could a module be used? Spencer has been looking into it. Martin is keen to keep looking into how feasible is it.

Lachlan doing some tests with some compiler flags off

Lachlan: ESM1.6 with CABLE included as a library within UM7, rather than as source code, bitwise reproduces the release build when optimisations are turned off i.e. release compiled with -O0 is identical to library build with -O0. Doing a 10 year run with optimisations turned back on with -O2, and comparing results to determine effect of optimisations on behaviour. First few years suggest runtime/year is about the same between builds, perhaps even slightly faster with the library, but most likely just natural variance.

Australian PFTs

Ian: Martin and Jhan are still looking into Australian PFTs.

Radiation frequency

Martin: Using hourly radiation (rather than 3 hourly) gives a better land surface energy balance but also has a larger effect on net surface SW than expected (even seems to affect something as basic as amount of Rayleigh scattering). Likely to have an unacceptably large effect on the overall surface energy budget. Ian and Martin think it needs to be parked.

Forcing update

Martin: no update on the forcing but still waiting on the ozone. Paul hasn’t seen it yet (July is a possibility).

Optimisation

Manodeep: 20 model years per day using 4 nodes per day (1000 Su). 5 nodes can get us up to 22 model years per day (1200 Su). Anton: is it harder to justify on a NCIMASS type application, Manodeep highlighted that he has tested 1-4 nodes as well and they all are more mpi efficient so there’s a balance. Tilo: 1kSu 20 years/d is similar to esm1.5 and would be happy with that (for production runs). Aidan: are you resource or time constrained? Tilo: both and storage. Martin would like to see both configs available. Aidan: highlighted current tests are using cice4 would be good to confirm that cice5 has similar performance. Rachel: we are generally limited by our ability to post-process, quality control and assess simulations.

General updates?

None.

46

Meeting minutes (29/07/2025)

Participants:
@cbull
@clairecarouge
@anton
@lachlanswhyborn
@dhb599
@inh599
@dougiesquire
@jemmajeffree
@manodeep
@paulleopardi
@RachelLaw
@rbeucher
@tiloz
@micael
@spencerwong
@tiloz
@Jhan
@Aidan
@alexnorton

Chair: @clairecarouge
Minutes: @cbull

The day’s standup consisted of discussing revisions and updates to the implementation of development threads for the upcoming configuration/build freeze so that the final spin up can be done. During the meeting a live editable version of the table from Coordinating development threads for next phase of control experiment was edited.

Following the meeting @cbull and @clairecarouge revised the table, re-ordering in terms of sequencing and priority. The new table is posted here.

47

Meeting minutes (5/08/2025)

Participants:
@cbull
@clairecarouge

@micael
@anton
@inh599

@jemmajeffree

@manodeep

@paulleopardi
@RachelLaw
@rbeucher

@tiloz
@spencerwong
@tiloz
@Jhan
@alexnorton

@pearseb

@Aidan

Chair: @clairecarouge
Minutes: @cbull

Project Coordination and Workflow

  • Claire gave an overview of the proposed forward protocol for folding in everyone’s development threads. @jhan is confirmed as the runner for integrating changes.
  • Claire emphasized the importance of coordination between contributors, runners, and reviewers.

Experiment Management

  • Discussion on whether to start new experiments or continue existing ones when applying changes.
    • Anton and Chris highlighted the need for new experiments when restart files are modified.
    • Ian stressed the importance of maintaining an accurate spin-up table for tracking changes.
    • Mixed approaches have been used so far; flexibility remains depending on the nature of changes.

Status Updates

  • Anton: Order 1 items are ready to go; Anton confirmed a 50-year test was run over the weekend (to decide on lprec value).
    • @anton and @Jhan will have an offline meeting to coordinate getting all the order 1 items going.
  • Ian raised concerns about long-term cooling trends in the deep ocean.
    • Pearse suggested Antarctic Bottom Water dynamics may be contributing.
    • Tilo and Rachel discussed whether this is a spin-up issue or a deeper energy imbalance.
    • Further investigation is needed, but changes to sea ice and vegetation may help.
  • Ian: still trying to understand the long-term change in temperature. We’re nearly at 2000 years and for the last 500 years the deep ocean has been cooling. Pearse: Matt and Pearse think that the AABW upwelling is affecting tracers at mid-latitudes. This may have been driven by the changes we made in southern ocean fluxes (change of distribution of fresh water has changed southern ocean overturning). Pearse is wondering if we need even more spin up time for this problem might resolve itself? Tilo: thinks Ian is concerned that there is an energy imbalance issue not a spin up issue. Rachel: thinks CICE5 coming in is a priority and need to see what impact that has. Rachel has suggested if Ian thinks energy imbalance could be canopy related then latest veg distribution has some reduction in trees. Pearse views the impact on the BGC and global ocean metrics as minor.
  • Paul: still waiting on the ozone and nitrogen forcings. Current guidance is that it was due in July and they are still running simulations. See link.
  • Spencer: wanted to check if anyone had any follow up thoughts on the water balance issue. Do we want to try some of the UM changes or stick with what we have now? Ian: would be interested to see how the salinity is affected by the changes? But in general, would like to see the changes go in. Spencer: has done a 50 year run but didn’t see any major changes so will consider running it a bit longer.
  • Vegetation and Australian PFT updates are progressing.
    • Alex confirmed restart creation; awaiting system availability.

Scheduling and Prioritization

  • Chris Bull suggested swapping Spencer and Alex’s items with Manodeep’s if needed.
    • Manodeep noted oneAPI changes need a week of testing.
    • Decision: maintain current order but allow flexibility based on readiness.

Technical and CI Updates

  • Harshula asked for any Spack issues; none raised.
  • Ian reported no conversion failures in recent runs. Aidan commented that nothing has changed infrastructure wise during that time.
48

Meeting Minutes (12/08/2025)

Present: @anton @clairecarouge @Jhan @Aidan @dhb599 @dougiesquire @manodeep @MartinDix @micael @rbeucher @spencerwong @tiloz @cbull

Arrived later: @alexnorton @harshula @micael @paulleopardi @rbeucher

Chair: @clairecarouge

Minutes: @cbull

Run crash

@Jhan had a run crash over the weekend. The crash is using the “August” version (e.g. includes CICE5, old classic compiler but have changed version). Anton, Dougie and Spencer are currently looking into it. Jhan hasn’t yet tried year 52, will try that. Spencer has found that it’s not a deterministic crash, four short runs, two of them crashed and two of them didn’t (@dougiesquire for the two that didn’t crash, are they the same? @spencerwong will check ). @MartinDix would like to look at one run that crashed and another that didn’t.

New build correctness system

@Aidan rolling out build correctness CI checks. Would like to clean up the branch names/defaults that are being used in the build here. E.g. @dougiesquire oasis3-mct can likely use the main branch, he will run a check. @anton cice5 would need a careful merge but is achievable – once this is done could use variants rather than specifying versions (@dougiesquire supports this idea and will look into it). @Aidan thinks it’s better done before production runs — not clear that it won’t change answers. @manodeep thinks it could be a good time to update oasis MCT version 5? @clairecarouge not sure if we have the bandwidth for that. @manodeep / @micael we are currently maintaining two versions of the build system to workaround some of the issues with the old system. @manodeep if it’s not answer changing then the update could be done at any time.

Australian PFTs

[@inh599 @Jhan @MartinDix @alexnorton please update this section!]

Code development to introduce the new Australian PFTs is complete (likely anyway) - final revision were implemented last week to the land-use change code - note that this change requires alterations to both the UM+CABLE codebase and the configurations as we need to also change values in some of the UM namelists irrespective of what happens inside CABLE.

Previous testing of the code - conceptually identical science configurations just with PFTs technically renumbered 2→12 and 4→13 - produced unexpected large differences. Some of this has been identified to be linked to an existing bug in the UM convection scheme (uninitialised variable) which we have opted to leave in. The rest is of equivalent magnitude to differences that arise due to internal variability - and so would be expected given the inherent numerics of reordering in the grid cell averaging routines.

At this point we ( @inh599, @Jhan and @alexnorton) believe the code+config is ready to be updated. @RachelLaw’s call as to whether to proceed. First drafts of Australian PFT parameter values (CABLE namelist and input files) and land cover maps are also ready for later testing/calibration.

Other ESM1.6 branches

@anton do we include wombat in the 4xCO2 and 1% CO2 branches? Yes but wouldn’t expect an oceanic response (@MartinDix) when WOMBAT is “uncoupled”. @tiloz previous versions of WOMBAT this was a non-issue because the tracers were handled differently. @dougiesquire thinks the new WOMBAT version will respond to the forcings but is unsure how the atmosphere will respond. Issue for 1pct BGC and RAD runs BGC and RAD coupled simulations · Issue #170 · ACCESS-NRI/access-esm1.6-configs · GitHub

Land use change

@alexnorton is it helpful to have a historical run with the land cover changes? (Aware that we’re still waiting on some of the cmip forcings.) @clairecarouge do we have the input data for that? @inh599 thinks esm1.6 amip historical could be possible but perhaps not the “full” historical period (starts in 1978). @RachelLaw is testing this currently but there is a gap in the forcing, is considering using cmip6 input forcings. @alexnorton is interested if folks think this is a worthwhile idea? @inh599 are you imagining coupled? @alexnorton yes but is constrained by what is practically possible. @inh599 would like to be clear on what the objective of doing this is. @clairecarouge configuration stability?

Other items

From @harshula : @Manodeep Sinha & @Micael Oliveira , The oneapi build workaround fix is now in the master branch of ACCESS-NRI’s oasis3-mct.

49

Meeting Minutes (19/08/2025)

Present: @clairecarouge, @spencerwong, @manodeep, @paulleopardi, @dougiesquire, @RachelLaw, @harshula, @jemmajeffree, @Aidan, @MartinDix, @rbeucher, @inh599, @anton, @pearseb, @sofarrell, @dhb599, @alexnorton

Chair: @clairecarouge
Minutes: @spencerwong

New experiment runner
As @Jhan will be on leave for a couple of weeks, it was decided to try and find someone to cover as the experiment runner, especially in light of the recent random crashes. @tiloz, @clairecarouge, and @RachelLaw will discuss further to find someone.

Wombat crashes
Random wombat crashes have been occurring at the start of simulations. @pearseb has looked into it. The crashes always occur at a particular routine, though there’s no indications of problems in the MOM restart data.

@MartinDix suggests that there could be something not being read correctly, and @manodeep mentions that there could be a race condition. @dhb599 has also had this error during test runs. Usually it resolves upon restarting, however in one case, restarting repeatedly reproduces the error. @dougiesquire will investigate more over the next few days. @spencerwong will test whether oneAPI has any effect.

Australian PFTs
@RachelLaw and @alexnorton are running several AMIP runs to test different parameterisations for the Australian PFTs. So far, these do not appear to cause major differences, and the changes in carbon fluxes are within the range of variability. More tests are being run to make sure that various metrics look sensible. LUC is also working.

Ocean temperature drift
@tiloz mentions that the priority should be on understanding the ocean temperature drift before bringing in further configuration changes, especially as the drift has increased when changing to CICE5.

@MartinDix is testing ocean albedo changes, and has done additional tests with the AprilSpinup configuration. Using the AprilSpinup configuration but removing the solar constant change results in a short drop in ocean temperature, but does not show any large long term drift.

@spencerwong will look into the proposed water balance changes. They will alter the global runoff, and hence will likely impact the iceberg melt.

Infrastructure changes
@Aidan notes that there are infrastructure changes ready to be made to the ESM1.6 model deployment repository, however it makes sense to wait until development is less active to make these changes, as they will require rebasing each open PR. We should let @Aidan know when it’s a good time to make these changes.

50

Meeting Minutes (26/08/2025)

Participants: @clairecarouge , @cbull , @RachelLaw @spencerwong @MartinDix , @manodeep , @tiloz , @Aidan ,@alexnorton, @harshula, @inh599, @jemmajeffree, @micael, @rbeucher, @sofarrell

Chair: @clairecarouge

Minutes: @cbull

Recent crashes

@spencerwong Testing different versions of mpi and compiler. Seems some versions of openmpi / wombat stop the crashes. @dougiesquire/@pearseb is looking into it further.

CICE crash, @spencerwong was talking to someone at NCI, upgrading openmpi helped there too. From crashes 1/100 to 1/700 years.

@dougiesquire suggestion: oneapi and openmpi 5 that we go that way as soon as possible (the wombat issue can be further investigated on the side). @manodeep is currently testing this combination / optimising.

Action: @manodeep will put the related PRs together.

Changing compiler flags is currently difficult

@paulleopardi , @lachlanswhyborn, @harshula has been asking about the Spack build. Current spack build does not allow one to change compiler flags for components that use the FCM build system (UM7 and GCOM) on the fly. Doing some scoping work to see if this can be modified. @lachlanswhyborn clarification: spack.yaml does in theory allow one to specify flags, however UM will silently ignore all these flags or architecture flags.

Ocean drift issue

@tiloz : discussion with many people on Friday. There were two options and one lesson that came out:

  • change parameters that influence freshwater fluxes in the polar regions but likely quite a limited impact;
  • change ocean albedo.
  • @RachelLaw lesson learned: we now understand why CICE5 rather than CICE4 has significant changes in ocean cooling, related to value of aicenmin(default is 1e-11) but we think we need something close to what it is now in CICE5 (2e-4) same as on UM7.3 side of code @sofarrell.

@MartinDix : update on changing ocean albedo (tweaked to cancel out solar constant changes – reduction in 4% of absolute value), from the start of the “June spin up”. Looks like it helped but not enough. Considering trying the same parameter change from the start of the “April spin up” (when the solar constant change was done @sofarrell ).

Large temporary output file from proc 207

Found when @RachelLaw and @inh599 were discussing a crash. Work output in processor 207 (the proc responsible for the rivers). @inh599 is logging saying that outputs are “not positive”. Possibly related to how values have been changed in inland water basins. One solution could be suppress the warning (the solutions are ok). @MartinDix : is it happening everywhere? @inh599 unsure, comes out in the work temp dir. @spencerwong: there is an upcoming change that will resolve this issue.

OASIS3MCT

Last week we talked about getting rid of esm1.5/esm1.6 branches in the build. Currently, we are using the esm1.5 branch for oasis. @dougiesquire did a test with the master branch and it’s bitwise reproducible. Note compiler flags will likely change when we start using the same branch across both models.

@manodeep: Upstream is version 5, currently we are using version 2. Quite a few years in between the two versions. In version 5, load balancing logging output is a welcome improvement and would help the optimisation work (better load balance).

@dougiesquire would ask that we do the upgrade in two steps: 1. switch to main branch, then do upstream update.

No objections from the room on these proposed changes.

Misc

  • @RachelLaw has updated the table to capture all the planned land updates and is looking to pull them together, a little difficult as @Jhan is on leave. @clairecarouge is happy to help with the build side of things. Related issue.
  • @paulleopardi forcings for ozone and nitrogen are now available. Stilll missing documentation on what’s inside them. Pending issue: files on Gadi are out of date and Paul needs a Gadi software update to get newer versions.
  • @RachelLaw : would be good to do a historical test (with CMIP6 forcing) since land-use change file now ready. @MartinDix is happy to set up configuration.

51

Meeting Minutes (02/09/2025)

Participants: @MartinDix, @spencerwong, @Aidan, @tiloz, @Jhan, @matthew.chamberlain, @manodeep, @anton, @micael, @clairecarouge, @dhb599, @alexnorton, @inh599, @sofarrell, @harshula, @pearseb, @paulleopardi

Ocean albedo and temperature drift

@MartinDix has done 50 year tests with the AprilSpinup configuration, scaling the ocean albedo by 0.96 and 0.92. 0.92 is much more effective at reducing the temperature drift. While 0.92 sounds like a drastic change, the resulting albedo is much closer to CM2 which used the Jin parameterisation. 0.92 also better matches the CERES data, which also uses the Jin parameterisation.

@MartinDix will see whether it’s possible to implement the Jin parameterisation in ESM1.6, as it may be preferable to the scaling approach.

Configuration for further ocean albedo tests

@MartinDix will test the albedo changes with an updated configuration. It was decided to include as many as the planned changes as possible for these tests, in order to use a configuration as close as possible to a final configuration. Along with the change to CICE5, further changes to be incorporated are:

A decision is needed on where to start the tests. @matthew.chamberlain notes that the ocean BGC reached a stable state during the JuneSpinup, and somewhere at the end of AprilSpinup or JuneSpinup could be a good starting point. @pearseb notes that later restarts with a more spun-up BGC state aren’t necessarily better, as physics changes make the completed spin up less useful.

It was decided to branch from two points in parallel. This will require CICE5 compatible restarts (@anton).

Mediteranean salinity issues were discussed, however it was decided they weren’t currently important for the global scale problems currently being focused on.

Wombat crash

@dougiesquire has fixed the intermittent wombat crashes, and created new executable and configuration releases. He can help if anyone wants to bring the fix into their existing simulations.

Openmpi5.0.8

@manodeep and NCI have solved an issue previously preventing us from using openmpi 5.0.8. The new version improves performance, putting the model on par, if not slightly better than ESM1.5.

AugustSpinup run

It was decided to stop the AugustSpinup simulation as we’ll be using an earlier restart.

Land changes

@RachelLaw and @alexnorton will work on the land changes in parallel. The changes are currently working, however @RachelLaw will investigate an issue where swapping between CN and CNP modes change meteorology when the feedbacks are switched off.

Next meeting

Next week’s meeting will be cancelled due to the workshop.

52

Shouldn’t date be 2/9/2025 ?

53

Thanks, fixed!

54

Date: September 16, 2025
Chair: @MartinDix
Participants: @anton

@MartinDix

@RachelLaw

@Jhan

@pearseb

@Aidan

@alexnorton

@dhb599

@dougiesquire

@harshula

@inh599

@manodeep

@micael

@rbeucher

@sofarrell

@spencerwong

@cbull

Minutes @cbull

Meeting focused on discussing these changes.

1. Vegetation Distribution & Australian PFTs

  • Vegetation distribution and historical land use maps (from 1850) are ready and tested.
  • Australian PFTs are integrated into the global map; parameterization is complete.
  • Rachel/Ian: reminder that the changes need to be implemented at the same

2. Ocean Albedo Drift & Scaling

  • Martin: newer albedo scheme in the ocean is having the opposite influence than what Martin expected (cm2 opposite behaviour!)
    • best option right now is the 0.92 (we have 50 years with that), considering 0.90 as well. Happy to start runs on p96.
  • Tilo: suggest halfway through/end of April spinup as a starting point for testing. Ideally including the other discussed changes (if that’s possible to do this week).
  • Tilo (sidepoint): Concerned that allocations have been downwards corrected for next quarter (Pearse can use any remaining allocations).

3. OASIS3-MCT Versioning

  • Harshula: Was there any further discussion around OASIS3 mct?
  • Current plan is for all builds to use the master branch (our fork). Version 5 is not answer changing so we can make this decision later.

4. Ozone Forcing

  • ESGF synchronization issues delaying ozone data download.

5. Iron Deposition & Wombat Light

  • Dougie: Iron deposition update is ready to go. Will likely include in the next batch of updates (will increase the amount of iron going into the ocean).
  • No current tweaks to wombat lite (Pearse).

6. ESM 1.6 Output CMOR-isation

  • Romain: ESM1.6 spin up has been partially cmor-ised. Romain interested in cmor-ising more of the data but wondering where to put it? (The goal is to compare it to esm1.5 run; lots of interest at the workshop). Martin suggested putting it on zv30. Rachel is thinking it might be worth holding off until we’ve implemented the latest batch of changes. Romain: could be some interest in checking the existing esm1.6 cmorisation (will check the status of this). Rachel: would be worth doing the full suite of cmip variables (Martin issue 164 hasn’t been actioned yet? Rachel: no, more compiling what’s needed).
55

Date: September 23, 2025
Chair: @clairecarouge
Participants: @anton, @clairecarouge, @RachelLaw, @Aidan, @alexnorton, @dougiesquire, @inh599, @manodeep, @MartinDix, @pearseb, @rbeucher, @sofarrell, @spencerwong, @Jhan, @cbull
Minutes: @cbull

@RachelLaw. candidate spin up runs? What should the test cases look like?
@clairecarouge, need to finalising the code base. As Rachel’s proposed tests will use one code base.

Finalising the code

Worked through the development thread.

@spencerwong, water balance, just waiting for final review of that code.

@manodeep, oneapi is all done and there is an open PR (@dougiesquire) to get it added to the config.

@anton. Has some CICE parameters that could be worth including in the spin up candidates. (aicenmin parameter affects ocean temperature; spin-ups with varied values proposed.)

Re: Mediterranean Bathymetry

  • Dropped due to lack of improvement; deep ocean drift prioritized.

Re: Forcing Files

  • Paul working on forcing suite with Martin.
  • Volcanic forcing update caused ~1% change in PI control; likely to be included.

@Jhan has a small transporation on tiles diagnostic change. Would like to try and get it into the next update (but not critical), @dougiesquire can help coordinate.

@sofarrell:

aicenmin that Anton mentioned is the only parameter thats the issue, we talked in Melbourne about seperating out the key code that will be impacting and testing that. I am not sure if Anton has on tm70 time to have tested that since. It was only a couple of years.

@anton: has fixed the diagnostics and should have some data to share in the next few days. Would require a code change (one liner).

Re: Coordination Strategy

@MartinDix. Albedo scaling to be added as a namelist variable.

candidates for spin up

@RachelLaw suggest having another meeting to coordinate.

@RachelLaw: @tiloz etc has regular team meeting tomorrow morning. All are welcome at 10am. @MartinDix would like to attend. @manodeep suggesting we have a new table for the different spinups (@clairecarouge agrees). @RachelLaw will work towards making a new table (similar to these perhaps?).

Other

Git & Cloning Issues

ACCESS-ESM1.6 configuration documentation.

57

Date: September 30, 2025
Chair: @clairecarouge
Participants: @Jhan, @MartinDix, @clairecarouge, @tiloz, @spencerwong, @Aidan, @dougiesquire, @manodeep, @inh599, @sofarrell, @harshula, @paulleopardi
Minutes: @spencerwong

Update on new spinup simulations

  • @Jhan is running two new spinups in parallel. These include all the priority 0 changes from the coordination table, including CICE5, water balance changes, oneAPI, new vegetation distribution and new PFTs.
  • Both simulations are using the same configuration with the 0.92 ocean albedo scaling. Aicenmin has been reset to puny and a new estimate for the add_lprec parameter is being used.
  • Both simulations show improvements in the global ocean temperature drift
  • We don’t plan to have any more answer changing code updates unless testing shows that they are necessary.

Further work

  • Ozone and nitrogen forcings will be updated in future. @paulleopardi, @MartinDix and @spencerwong to meet to discuss how to produce a new ozone ancillary.
  • Other configurations (e.g. emissions driven, 1pct CO2, 4xCO2) to be brought up to date – however we won’t run new tests with these until we have a more stable starting point.
  • @MartinDix, @paulleopardi, @spencerwong to set up an initial historical simulation using CMIP6 ozone. @inh599 notes that wood harvest may need to be updated to work with the Australian PFTs.
  • Output names and metadata: @inh599 notes that it’s a good time to make sure that all output names, units etc that appear in the output file are correct. @Jhan is working on changes to the STASHmaster, however changes to um2nc will be needed for updating the postprocessed netCDF files. @MartinDix to share the um2nc ticket.
  • @spencerwong to update the netCDF conversion to remove the extra netCDF directory level. This should be incorporated between the current spinups and the official PI control run.
  • @Aidan is working on adding metadata to model outputs.
  • Configuration documentation also needs to be written

Changes to standup meetings

  • With the code in its final iteration, it’s less crucial for everyone to attend the meetings every week. Feel free to skip meetings if you think the discussion won’t be relevant to you.
58

Date: Oct 7, 2025
Chair: @MartinDix
Participants: Martin

Anton

Dougie

Rachel

Romain

Spencer

Siobhan

Tilo

Chris B

Alex Norton

Aidan

Paul

Jhan
Minutes: @cbull

Martin gave an update on control run candidates ( Global ocean temperature drift · Issue #188 · ACCESS-NRI/access-esm1.6-configs · GitHub )

Tilo: would like to keep all three going. B run looks promising.

Martin: a/b/c configs only differ in the restart files.

Rachel: has a few concerns about carbon. On the land, effectively photosynthesis, seems to come in before some of the more recent changes. Something happened between the August spin up and PR-129 (Claire was doing these) – code changes that went during that period don’t seem to explain the land carbon changes. Martin: would assistance be helpful for unravelling which code changes were made? Rachel: Claire was managing this but help would be useful. Changes seem to be occurring with PFTs in the tropics (Martin: not obvious what we would have done to affect that). Currently not documented in an issue. Martin and Rachel agreed that putting some figures in an issue would be a useful next step.

On the ocean carbon side (Rachel), seem to have more of a change ? (iron deposition to blame?).

Jhan has been looking at the UM7 repository (ACCESS-NRI repo), would like to see more of the open issues resolved to be consistent with the approach taken for AM3. Martin: some of those issues are more ‘thought bubbles’ could be deferred.

Tilo: Are the emission driven forcings up to date? Could we possibly start a perturbation simulation soon-ish? Spencer: There was a PR for updating the emissions driven config. Tilo: we do need an emissions driven control as well, so would be good to parralise. Spencer: where to start? Tilo: probably at the end of the B case.

Paul:

1. Volcanic, would need to talk to spencer. Might be that the namelist wasn’t filled out to the end.

2. Still working on the ozone forcing. Qn: what is supposed to happen when it’s ready? When will it be incorporated? Martin: ozone differences look pretty small, so could likely put them in half way through a run. Paul: nitrogen is still coming.

3. The emissions driven stuff, haven’t looked at them yet. Martin: The PI-control has 0 emissions (so nothing needed there but a zero field). Martin: Actual emissions driven, haven’t looked at those fields.

Rachel: there are some idealised runs we need to set up at some point too. Namely: flat 10, 10 petagrams of carbon per year (constant) – (also: flat 10 followed by 0 emissions, flat 10 CDR reduces emissions linearly and then -10 constant petagrams of carbon per year); these 3 are part of the fast track experiments. We had a script for setting this up in esm1.5 (there’s also a set of tools at access-nri). Martin: would be good to set these up as well. Tilo: relevant past discussion here

59

Meeting minutes 14/10/2025

Participants:

Chris Bull
Anton
Claire
Martin
Ian
Rachel
Siobhan
Tilo
Jhan

Chair: @clairecarouge Minutes: @cbull

Claire described that GitHub management has changed, please let us know if you have lost write access to any repos. Can post on the forum, or in the community repo case open an issue.

Spinup. Martin gave an update via this issue. @matthew.chamberlain did have a look and shared some plots at a recent team meeting, he showed a warmer surface ocean and cooler deep ocean. Siobhan offered to have a look at ocean diagnostics. Rachel: Pearse and Dave B are currently both away.

There was a fix that went into UM7. Rachel: had an unexplained change in the land carbon fluxes bug (from last week). Rachel since opened a GitHub issue and we found a bug in one of the changes we made. Bug has been corrected (in UM7). Now run about 70 years in parallel. The plan is to swap over all candidate runs to the corrected executable.

Ian concern: we currently have mismatched clocks from using different restarts. Martin: the a/b/c runs all have consistent years via fiddling with the restarts. Anton: the dev-preindustrial branch on GitHub still has mis-matched years.

Paul: both the ozone and nitrogen forcings are available. In both cases, the original published forcings had problems so they have been updated. Paul has an issue that the inputs are given on pressure layers. Paul is unsure how to proceed and thinking he will ask for help on the forum. Martin: the model can save outputs on pressure levels. Martin and Paul can discuss. Martin thinks the ozone from CMIP6 isn’t that different so likely won’t lead to substantial changes (particularly for pre-industrial). Rachel: suggested we need to be careful about units for nitrogen deposition for cable (note Rachel thinks offline and online cable have different unit requirements!).

Chris: has mentioned that we’re thinking about an esm1.6 release and config’ docs/how to run esm1.6 and write up. Ian: suggested it would be good to get input from a community person on what is relevant to them from esm1.5. Chris: suggested focusing in the first instance on what technical information is helpful for us in terms of writing papers/remembering thinking behind parameter choices etc. Rachel: suggested it would be helpful to have an outline of the structure of the docs and offered to help brainstorm a first version with Claire and Chris. Ian is interested too but it will have to wait till next month due to more immediate priorities. Chris suggested we proceed with the three people and then other folks are welcome to join when they can. Will be discussed at the leadership meeting this week as well.

Paul: Have we collected enough information to write the updated versions of the relevant parts of the previous paper: https://www.publish.csiro.au/es/fulltext/ES21031 (e.g. model description, forcings?) Rachel: . How many papers we want to write and which ones? But yes that kind of paper is a sensible target to aim for.

Chris: CM3 evaluation training at 1pm this afternoon. Feel free to come along.

60

Meeting minutes (21/10/2025)

Participants: @anton @inh599 @MartinDix @sofarrell @tiloz @adfraser @clairecarouge @manodeep @RachelLaw @Jhan @cbull

Chair: @clairecarouge

Minutes: @cbull

Claire: Perhaps a small attendance due to eResearch?

Confusion of water flux signs in cice/um

Anton: Been looking at evaporation diagnostics in cice/um (surface_snow_and_ice_sublimation_flux: ['fld_s03i298']). In both cases, they are not sign definite. Martin thinks it might include dew. Ian suggested tracking back to look is it evaporation or is it flux of water into/out the atmosphere (which allows both signs), be careful about phases too! Ian: thinking about surface_snow_and_ice_sublimation_flux, it could be: snow pack → atmosphere or depositon of frozen ice crystals onto snow pack (i.e. allowing both signs). In general (Ian) most of the fluxes in these context can be either sign.

Martin: note esm1.5 had sublimation wrong (in terms of units Ian)

spin-up updates: which experiment to choose?

Update post

Ian: year 30 of spin up experiment B has done something strange.

Tilo: have you considered re-running? Ian: yes, it’s possible, we’d need to re-run year 20-30 due to restart file availability. Tilo: would like us to re-consider keeping more restarts in the future. Jhan/Anton: it’s possible to change this in the config.yaml as one of the runtime options. Tilo: yes, would be helpful with future science applications, probably less important for the current run but helpful in the future (Tilo will consider opening an issue). FYI: the model is restart reproducible but not reproducible across different restart boundaries..

Test B appears to be more stable. Claire: do we need longer? Do we want to make a decision? More analysis of the ocean output? Siobhan: been looking at September A/C, both are looking at least 1 degree cooler in deep ocean . Martin: did some hovmollers, showing warmer southern ocean/cooler north atlantic (80-100 variability Labrador and Greenland sea), once the albedo change is switched on pacific doesn’t yet have an emerging signal but that would have expected as timescales there are 1500-2000 years. Next up is to look at September B but expecting it to look worse. Similarly for October across all runs. Siobhan would be curious about what’s happening with BGC (no one on the call). Rachel: would we expect all of them to eventually equilibrate at the same point? Siobhan: not sure at this point or ready to choose at this point.

CICE updates

@anton has been working on CICE diagnostics but he’s not sure of the deadline? @sofarrell has gone through the output and the code and it’s looking okay. @anton still wants to fix temperature, @sofarrell has looked into this from BL99 and mushy but is unsure how to apply similar fixes to zero layer model.

Rachel: on timing. We’re thinking early November we need to switch across to an “official” PI control and that would give us ~500 years. Tilo: that assumes all the other things are finalised (CABLE as a library, data request etc.)

Planned updates before the official pi control

@clairecarouge suggested a summary of required changes would be helpful, first try at a list:

We have <2 weeks to do this!

Re: next week’s standup. Next week people have some training/NESP commitments, we can move the standup if that’s useful? Just let @clairecarouge know.

@RachelLaw @clairecarouge @cbull are meeting at 3pm today to talk docs, all welcome. Get in touch with @cbull for an invite.