Meeting Minutes (26/08/2025)
Participants: @clairecarouge , @cbull , @RachelLaw @spencerwong @MartinDix , @manodeep , @tiloz , @Aidan ,@alexnorton, @harshula, @inh599, @jemmajeffree, @micael, @rbeucher, @sofarrell
Chair: @clairecarouge
Minutes: @cbull
Recent crashes
@spencerwong Testing different versions of mpi and compiler. Seems some versions of openmpi / wombat stop the crashes. @dougiesquire/@pearseb is looking into it further.
CICE crash, @spencerwong was talking to someone at NCI, upgrading openmpi helped there too. From crashes 1/100 to 1/700 years.
@dougiesquire suggestion: oneapi and openmpi 5 that we go that way as soon as possible (the wombat issue can be further investigated on the side). @manodeep is currently testing this combination / optimising.
Action: @manodeep will put the related PRs together.
Changing compiler flags is currently difficult
@paulleopardi , @lachlanswhyborn, @harshula has been asking about the Spack build. Current spack build does not allow one to change compiler flags for components that use the FCM build system (UM7 and GCOM) on the fly. Doing some scoping work to see if this can be modified. @lachlanswhyborn clarification: spack.yaml does in theory allow one to specify flags, however UM will silently ignore all these flags or architecture flags.
Ocean drift issue
@tiloz : discussion with many people on Friday. There were two options and one lesson that came out:
- change parameters that influence freshwater fluxes in the polar regions but likely quite a limited impact;
- change ocean albedo.
- @RachelLaw lesson learned: we now understand why CICE5 rather than CICE4 has significant changes in ocean cooling, related to value of
aicenmin(default is 1e-11) but we think we need something close to what it is now in CICE5 (2e-4) same as on UM7.3 side of code @sofarrell.
@MartinDix : update on changing ocean albedo (tweaked to cancel out solar constant changes – reduction in 4% of absolute value), from the start of the “June spin up”. Looks like it helped but not enough. Considering trying the same parameter change from the start of the “April spin up” (when the solar constant change was done @sofarrell ).
Large temporary output file from proc 207
Found when @RachelLaw and @inh599 were discussing a crash. Work output in processor 207 (the proc responsible for the rivers). @inh599 is logging saying that outputs are “not positive”. Possibly related to how values have been changed in inland water basins. One solution could be suppress the warning (the solutions are ok). @MartinDix : is it happening everywhere? @inh599 unsure, comes out in the work temp dir. @spencerwong: there is an upcoming change that will resolve this issue.
OASIS3MCT
Last week we talked about getting rid of esm1.5/esm1.6 branches in the build. Currently, we are using the esm1.5 branch for oasis. @dougiesquire did a test with the master branch and it’s bitwise reproducible. Note compiler flags will likely change when we start using the same branch across both models.
@manodeep: Upstream is version 5, currently we are using version 2. Quite a few years in between the two versions. In version 5, load balancing logging output is a welcome improvement and would help the optimisation work (better load balance).
@dougiesquire would ask that we do the upgrade in two steps: 1. switch to main branch, then do upstream update.
No objections from the room on these proposed changes.
Misc
- @RachelLaw has updated the table to capture all the planned land updates and is looking to pull them together, a little difficult as @Jhan is on leave. @clairecarouge is happy to help with the build side of things. Related issue.
- @paulleopardi forcings for ozone and nitrogen are now available. Stilll missing documentation on what’s inside them. Pending issue: files on Gadi are out of date and Paul needs a Gadi software update to get newer versions.
- @RachelLaw : would be good to do a historical test (with CMIP6 forcing) since land-use change file now ready. @MartinDix is happy to set up configuration.