CSIRO - ACCESS-NRI standup minutes

CMIP7 CMIP7 development
21

Meeting Minutes
Date: February 18, 2025
Attendees: Chris Bull, Anton, Pearse Buchanan, Claire Carouge, Ian, Tilo Ziehn, Aidan Heerdegen, Dave Bi, Martin Dix, Manodeep Sinha, Micael Oliveira, Tammas, Dougie, @paulleopardi
Minutes: @cbull

Agenda:

  • Project Board Review
  • Timelines for Work
  • Model Validation and Spin-Up Process
  • Ancillary Files for Production Runs

Discussion Points:
Project Board Review:

Chris Bull asked if there were any issues or blockers regarding the project board.
Pearse Buchanan noted a few ocean things were labelled in progress even though they are complete, he will update. Claire Carouge emphasized the importance of closing pull requests to reflect completed tasks.

Claire Carouge suggested (next action for everyone) reviewing the number of ā€œTodoā€ issues and setting priorities or removing from project board if not within scope.

Timelines for Work:

Chris Bull inquired about the timelines for work and the version of the model to be used for production runs in July. Tilo Ziehn explained the current status of the ocean spin-up and the integration of land model updates.

The goal is to use the long simulation runs by Pearse Buchanan for the spin-up (currently at year 543). There is not a target number of years (rather equilibrated metrics) but currently the model gets around ~25 years per day.

Chris Bull highlighted the importance of coordinating work tasks to meet the July deadline for fast-track production runs.

Tilo Ziehn discussed the plan to integrate model updates and run the spin-up continuously with CABLE being added when available (new cable version currently being tested in ESM1.5). The aim is to have a stable model configuration by the end of June. (Timing of the integration of CICE changes was not discussed.) This is not the best practice ā€˜continuousā€™ spinup that the CMIP protocol requires to be publishable.

Side point: Ian asked Tilo about recent CABLE changes and Ian offered to have a look at them, Tilo Ziehn said the run is available: /scratch/p66/txz599/archive/PI-restart/history/atm which is the current test run with the new vegetation fractions. Still raw binary output, but can extract variables in netcdf as required.

NB: Manodeep/software transformation team are able to do their best work when the model configuration is finalised as soon as possible.

Model Validation and Spin-Up Process:

Aidan Heerdegen proposed automating model validation tests to reduce manual inspection. Pearse and Aidan discussed what that could look like and that itā€™s quite flexible.
Dave mentioned the availability of automatic tools for model performance evaluation. Chris suggested Dave is free to share more about that at future stand up sessions.

Ancillary Files for Production Runs:

Paul raised a question about the ancillary files needed for the production runs:

which CMIP7 forcings for ESM1.6 need to be ready for the spinup in July. Pre-industrial, historical, other?

As a list of priorities, Tilo and Ian agreed it is: picontrol, 4x1pct, historical. Tilo Ziehn emphasized the importance of having the necessary ancillary files ready for the July production runs.

Action Items:

  • Review and prioritize outstanding Todo issues on the project board (All).
  • Continue the ocean spin-up and integrate land model updates (Tilo Ziehn, Pearse Buchanan).
  • Consider development and implementation of automated model validation tests (Aidan Heerdegen, Dave, Pearse).
  • Discuss timeline for CICE changes in ESM1.6 at next weekā€™s standup.

Call for other business regarding ocean and land teams:
No items.

Next Meeting: 25/02/2025. Claire will facilitate.

22

Thanks! Please add my name to the attendees, since I am mentioned in the minutes.

23

Thanks @paulleopardi, now added (note you can edit as these are wiki posts). Unfortunately AI seems to get this wrong every time ā€“ so need to remember.

24

Thanks!

25

Meeting Minutes
Date: February 25, 2025
Attendees: @anton , @clairecarouge , @inh599 , @Jhan , @rbeucher , @Aidan , @arnoldsu , @ben , @harshula , @manodeep , @MartinDix , @micael , @paulleopardi , @pearseb , @spencerwong , @tiloz , @cbull

CICE5 implementation timeline concerns
Claire and Tilo discussed the timeline for CICE 5 implementation, noting that it might not be ready in time for the production runs. They agreed getting more visibility of the work soon is critical. Getting the code in GitHub by the end of April at the latest is essential for any hope to integrate CICE5 in ESM1.6.
Tilo and Chris and others concerned will discuss with Dave and Shioban when Dave comes back from leave.

Problem with the netcdf conversion for UM files
Ian and Jhan noticed the netcdf conversion is sometimes failing. This causes issues for quick evaluation of runs with standard scripts. Aidan and Spencer are aware of the issue and looking into it.

Cmorisation of outputs
Romain is now attending the stand-ups. He is working with Arnold on preparing the cmorisation for CMIP7.
He asked for clarity on the raw output format that will be produced. He will also document all the calculations done on variables for review by the group.

New executable for long runs
Ian said he and Jhan are consolidating several parallel developments in a single source code. These changes also include changes in the configuration and in payu.
In about a week, we are aiming to replace the executable for the long running simulation with this new source code as well as bringing in some changes on the ocean side. The timeline depends on how many sanity checks we want to do beforehand.
Manodeep mentioned if we change the executable it might be good to run on Sapphire Rapids and if possible to compile with the OneAPI compiler. Moving to Sapphire Rapids show a 20% reduction in CPU time and a reduction in cost.
Spencer mentioned adding more CPUs for the atmosphere didnā€™t show the same improvements as for ESM1.5. This is because in ESM1.6 the atmosphere is waiting on the ocean and not the other way around.

Forcings creation
Paul now has access to scripts used for CMIP6. He needs additional explanations from Tilo or Martin to understand how the scripts work and how to modify them for CMIP7.

Actions

  • Tilo to organise a meeting for CICE5 when Dave comes back.
  • Aidan and Spencer to investigate the collation and conversion issues
  • Ian and Jhan to consolidate the land developments into a single source code and configuration.

Next meeting
March 4th, 2025. Chris to facilitate.

1 Like
26

Meeting minutes
Date: March 4th 2025
@Jhan, @cbull, @harshula, @manodeep, @RachelLaw, @MartinDix, @tiloz, @micael, @anton, @spencerwong, @paulleopardi, @inh599, @clairecarouge, @ben, @Aidan, @dougiesquire

Agenda Items

  • Chris Bull no pre-existing agenda items.

Communication Update

  • Claire Carouge highlighted the move of our ESM1.6 communications from a hidden message to a public topic.
  • Chris Bull clarified the renaming of the thread and encouraged attendees to watch the thread for notifications (click bell down the bottom).

Project Board Review

  • Chris Bull reviewed the ESM 1.6 project board and invited attendees to mention any blockers or issues.
  • No issues / blockers were mentioned.

Experiment Documentation

  • Chris Bull emphasized the importance of documenting experiments, especially those used for production runs in July. This process has begun already, thanks @spencerwong and @pearseb
  • Spencer Wong discussed the creation of a GitHub repository for experiment details and provided instructions for pushing experiment branches.
  • Specifically this branch naming convention <GH-user-name>-<dev/test>-<YYYYMMDD>-<counter> and the use of GH issues for any descriptors. Please see README for further instructions.
  • @pearseb not here today but please add experiment for current spin up experiments to this repoā€™.
  • Note this is a different GH organisation so youā€™ll need to get in touch with @clairecarouge / @spencerwong to get access.

General Matters

  • Claire Carouge raised an issue about building and running ESM 1.6, mentioning a runtime error related to generic tracers.
  • Dougie Squire and Harshula discussed potential issues with the build system and configurations and the need for updates on both.
  • Dougie Squire wanted to know which config branch to focus on, this one

Data Management Plan

  • Ian raised a concern about data storage and the potential for data purging on scratch storage.
  • Rachel Law and Tilo Ziehn discussed the need for a data management plan, especially for spin-up and production runs. Only a subset is probably practical to keep on disk the rest can go on tape.
  • Paul Leopardi requested a copy of Chloeā€™s original data plan for reference.
  • No one is currently assigned to write a Data Management Plan (afterthought: @cbull will ask @ClareCR about next steps?)
  • @Aidan payu has in-built sync support. If weā€™re in danger of losing important information it should be turned on

Compiler and MPI Updates

  • Harshula expressed the need to use a newer compiler and OpenMPI version for future test runs. Namely openmpi: 4.1.5 / ifort: 2021.10.0. See PR. When is the best time to roll out these changes?
  • No one followed up at standup. Chris would follow up offline.

Action Items

  • Chris Bull to follow up on the sequencing of changes related to the new compiler and MPI version.
  • Rachel Law to provide Paul Leopardi with a copy of Chloeā€™s data plan.
  • Dougie Squire to review and update ESM1.6 configuration to ensure compatibility.

Closing Remarks

  • Dougie Squire Looks like I wonā€™t have time to raise my hand. I want to update the ocean grid, which will also entail updating the ice grid and remapping weights: link
  • Chris Bull Out of time today, please send agenda items in advance such that we can plan for time as appropriate
  • Chris Bull thanked attendees and closed the meeting. @clairecarouge is hosting next week.
27

Meeting minutes
Date: March 11th, 2025

@Jhan , @cbull , @clairecarouge , @harshula , @manodeep , @RachelLaw , @MartinDix , @tiloz , @arnoldsu , @spencerwong , @pearseb , @paulleopardi , @rbeucher , @inh599 , @ben , @Aidan , @dougiesquire , @tammasloughran

OneAPI new executable

  • Manodeep discussed the move to OneAPI and whether to apply the change in one go or incrementally.
  • Martin expressed developers donā€™t need access to intermediate steps and switching to OneAPI in one go would be fine.
  • Aidan and Harshula mentioned the pre-release infrastructure allows for several executables to be created within a single PR. We should take advantage of this so that, during the development phase of the OneAPI support, changes to the compiler can be done incrementally and tested.
  • Dougie mentioned changes to the configuration might be needed in step with changes to the compiler. Currently, ACCESS-OM2 executable using OneAPI crashes the model. More investigation is needed to understand where it comes from.
  • Manodeep asks if there are tools to compare results of 2 configurations. Dougie said there are and will discuss this further with Manodeep.

Software update for ESM1.6 build

  • Harshula mentioned Dougie has a software update PR for the ESM1.6 build that we should finish first and use as a base for the OneAPI work.
  • Ian: there is a blocker here as there might not be a UM version everyone would be happy with. The latest version shows questionable results in a test run. A previous version is problematic as well because it does not have restart reproducibility. This needs a scientific discussion at CSIRO to decide how to fix the issue. [INH edit: All previous ESM1.6 versions do not have run-to-run reproducibility, ESM1.6 will never have restart reproducibility. It is the run-to-run reproducibility that is necessary.]
  • Tilo agreed we need to solve the question of the executable for the spin up quickly and it will need a meeting of the parties involved.
  • Rachel said we might be able to create an updated ESM1.6 build that is compatible with the configuration to enable the compiler work to move forward. And work on solving the issue with the UM version for the spin up separately.

Sharing experiments with the community

  • Chris said he has a meeting with Pearse, Spencer and Dougie to explain the workflow on how to share ESM1.6 experiments on the community repository. Anyone interested is welcome to attend.

Ocean grid update

  • Dougie asked whether we should update the ocean grid to avoid supporting a decade-old grid. This could simplify the support for the cmorisation. However, it would require changes to a number of tools and the remapping weights between the atmosphere and ocean grids.
  • Rachel had a concern about the land mask. Dougie and Martin assured the land mask would not change sufficiently (rounding errors) to be a concern.
  • Paul said the change will have an impact on the preparation of the CMIP7 forcings for the ocean.
  • Martin and Aidan agreed it would be good to update the ocean and sea-ice grids. Using an old grid format will make it much more difficult for Model Evaluation to support.

Action Items

  • Ian and Tilo to initiate meeting on solving the issues with the latest UM7 version with interested parties
  • Dougie to continue work on updated ESM1.6 build if possible
  • Dougie to initiate work on updating the ocean and sea ice grids.

Next Meeting

Tuesday, March 18th. @clairecarouge is hosting next week (@cbull is an apology).

1 Like
28

Meeting minutes
Date: March 18th, 2025

@paulleopardi , @MartinDix , @rbeucher , @manodeep , @clairecarouge , @tiloz , @RachelLaw , @anton , @spencerwong , @tammasloughran , @dougiesquire , @inh599 , @harshula , @micael , @Aidan , @dhb599 , @pearseb , @arnoldsu , @Jhan

CMIP7 forcings
We need a written description of the processing applied to the forcings See what was produced for CMIP6: https://www.publish.csiro.au/ES/ES21031 (section 2.3)

Paul to open a topic on the forum to gather input on the treatment of the forcings for CMIP6 and what needs to be done for CMIP7.

We will introduce the CMIP7-based forcings when they are available to use in ESM1.6

New spinup configuration
Temperature drift
New tests havenā€™t identified the cause of the drift. For the moment, tests seem to point to the calculation of canopy%epot but itā€™s still preliminary.
Need to review the whole setup of the simulation, Aidan and Spencer are happy to help with learning how to use payu effectively.

Outputs Collation
To solve recurring issues in the file collation after the simulation, it was decided to write the ocean outputs in a single file directly. This comes at the cost of 5 min in walltime. There is a bug with some MOM5 diagnostics if output to a single file, this wasnā€™t deemed major.

Sapphire Rapids
Changes are done and ready for people to use. People would need to rebase their development branches, contact Spencer. The change brings a 20min improvement in walltime.

PI restart with new veg. distribution
Some of the values in the file are not internally consistent. Problems disappear after a few years of running. Will pick up a restart from later in the simulation to remove these effects.

Question from Lachlan about the new for values for the previous year, Rachel replied we donā€™t need the previous year for the spinup phase.

Data management plan:
Peter Dobrohotoff looking into it but not attending the standups. It might be best to discuss it at the leadership meetings.
We also need to look into the output profiles we are using: it seems we are saving more data than we need. Nobody earmarked to look into it yet.

Non-determinism with changes in processor decomposition
A problem was identified that took about 6 months to appear.
Solution: have a cron test that runs every week to pick up when the model becomes non-deterministic. Aidan says the machinery is already here to make this happen (see issue).
Generally, Dougie says we need to run repro tests more routinely. Repro testing is installed in the access-esm1.6-config repository, people need to create pull requests in the configuration repository to trigger these.

Switch to OneAPI
Manodeep says configurations with OneAPI are ready, question about the timing for the switch of compiler.
Martin and Tilo: next spinup setting to run with ifort first. It will be good to be able to run a long simulation (~100 years) with both ifort and OneAPI to assess the effect on the climate before a definitive switch.

29

Meeting minutes
Date: March 25th, 2025

@tiloz , @clairecarouge, @RachelLaw , @manodeep , @MartinDix , @micael , @spencerwong , @pearseb , @ben , @harshula , @tammasloughran , @Aidan , @anton , @Jhan , @inh599, @dougiesquire (late)

Martin: Run with the new, reduced solar constant using Jhanā€™s latest config. Shows a cool down as expected but nothing dramatic so far.

Harshula: Openmpi 4.1.5 update - no effects on results and negligeable effect on performance.

Jhan: New couple of simulations using latest version of UM7, one with epot change and one without the epot change. Get the same results for both tests, still looking for why there is no difference. Ian: the 2 new test runs show good results compared to Pearse long run with very minor changes to the average ocean temperature.

Forcings: volcanic changes will be very small. Looking at the aerosols next.

ILAMB update: Romain gave an update at the CMIP7 Leadership meeting last Friday. He expects this to be ready in 2-3 weeks. Claire to contact Rhaegar to keep track of this activity.

Compiler update: When trying a simulation from a build with an updated ifort compiler, Harshula and Dougie had similar performance but different results. The next step is unclear. Question from Claire whether it is useful to investigate further considering the plans to change to OneAPI.

Walltime: Jhan has 20 years/day using Sapphire Rapids. Ocean outputs to one file per variable which would increase the walltime slightly. But Jhan noticed problems with the atmosphere conversion. Spencer to have a look.

Pearse: Update to the ocean BGC parameters to come. Long run at year 900.

Dougie: no progress on the new grid.

Dave: CICE5 still under way. Issue with initialisation, trying to resolve this.

Ian: the land group is trying to add some Australian plants in the model. New feature to come.

Rachel: Farewell to Tammas, finishing his post-doc this week

Action:

  • Make sure your build PR on ACCESS-ESM1.6 repository are rebased onto the latest main branch. Talk to @harshula if you need help or more information.
30

Meeting Minutes

Date: 01/04/2025
Attendees: @paulleopardi, @clairecarouge @micael @Jhan @tiloz @inh599 @spencerwong @manodeep @cbull @MartinDix @rbeucher @pearseb @dhb599 @Aidan @dougiesquire @anton @ben
Minutes: @cbull

Martin has a new run CMIP7 solar constant

@MartinDixā€™s new run has 0.3-0.4 degree cooling in the global mean. Will post some new Figures on the thread.

Paulā€™s update on CMIP7 forcing files

@paulleopardi has been working on the CMIP7 pre-industrial forcing files for use in ESM1.6. See issues for current challenges and wiki for current understanding of ESM1.5 cmip6 workflow.

Discussion on around which version of the land updates to merge into the current spin-up run

  • Ian raised concerns about the combination of land code changes merging into @pearseb existing pre-industrial run. At the moment they ( @inh599 @tiloz ) have some unexplained results and a lack of confidence that the build / config system is actually running the code they expect. Considering parking the current line of work due to time constraints and relative importance.
  • Claire Carouge suggested involving someone more familiar with the build system to fix issues rather than parking them.
  • Dougie Squire and Jhan emphasized the importance of tracing builds and understanding the changes.
  • Dougie/Claire/Chris: up to you as to how much time you have to persue it but it really helps if the build/config is on GitHub (example). Sounds like an issue worth getting to the bottom of. Next action: @Jhan to share the relevant builds on forum or GitHub. @Aidan: feel free to ask for reviews of payu/config use and also helpful if you can put plots on GitHub issues.

Further Land Team Updates

  • Ian mentioned progress in updating the ESM with new Australian vegetation PFTs (e.g. spinifex, eucalyptus), though it may be more complex than initially thought.
  • Claire Carouge gave an update on ILAMB. Rhaegar is now able to run ILAMB on ESM1.6 outputs. He will put together a workflow so anyone can use ILAMB with ESM1.6 outputs.

Software transformation team and MED team

  • Manodeep Sinha reported on the rerun of tests due to a non-deterministic bug in the UM which is being looked into in AMIP, confirming deterministic results across cascade lake/sapphire rapids. Looking to merge into esm.
  • Romain been using lite version of MOPPER to create CMORā€™isation of output. Will post on the forum about next steps.

Ocean Team Updates

  • Pearse Buchanan has been continuing the long runs. Branched off main run at about ~1000 years because there was a slight change, now no longer considering burial of matter through sediments. No longer an open boundary at the bottom for the carbon cycle ā€“ seems unsurprisingly thatā€™s bringing things closer to equilibrium. There will be another ecosystem component update soon (Pearse/Dougie hoping to merge wombat sinking PR today ā€“ slight answer change). Chris requested that the forum documentation and config repo get updated accordingly.
  • Dave has created a CICE5 branch for ESM1.6, put all of his updates of the modules there. Dave has had some trouble compiling. Anton has tried compiling and it looked like it built successfully (still need to check that the executable exists) ā€“ will look at it more closely today.
31

Meeting minutes

Date: 08/04/2025
Attendees: Romain, Claire, Harshula, Martin, Tilo, Michael, Manodeep, Jhan, Dougie, Dave, Aidan, Ian, Lachlan, Spencer, Pearse, Paul

Cascade Lake non-reproducibility
Manodeep: Configs now run on SapphireRapids (~20% faster). PI runs non-reproducible on CascadeLake cores when changing atmosphere layout. With the same ocean layout but different atmosphere layout. AMIP runs with any atmosphere layout are deterministic.

Micael and Martin agree it is not worth chasing further but important to document.

New WOMBAT-lite
@dougiesquire and @pearseb have released a new version of WOMBAT-lite on main. Can there be a new ESM1.6 build release using that update? It is almost identical to what Pearse has used in the spin up run. @dougiesquire cmake build stuff is still not in main, so could either release from dev tag or try to get that in. @Aidan the latter would be better. @harshula to organise doing this asap.
It has also implications for the new spin up configuration we want to put in place. The ocean restart will need some manual change in the deep ocean for the ocean BGC to use the new WOMBAT-lite. (@pearseb to do).
When do we want to start the new spin up? @tiloz does not want to wait too much as there are more changes coming. It might be good to start before Easter so we can analyse and fix some problems before it runs through Easter time.

Decision: we shouldnā€™t wait too long to the new WOMBAT-lite. If it can come in today or about, it can go in the new spin up. Otherwise, it will get integrated later on.

Spin ups and tests

@tiloz says Pearseā€™s current spin up can still run in parallel to the new spin up so Pearse can test changes in the ocean, sea ice and BGC. Same approach to be used if we have new changes focused on the land later on.

Salinity decrease
@dhb599 Salinity decrease, have tests been done with a parameter change to solve this? @pearseb Daveā€™s suggestion seems to solve the decreasing salinity. @inh599 says we may want to undo that because new land changes that are coming in are slightly increasing the salinity. The parameter is a tunable parameter so can be modified depending on what an experiment produces.

CICE5 technically working
@dhb599 says CICE5 is now working technically in ESM1.5. Havenā€™t checked results yet. @anton and Siobhan need to analyse it.

32

Meeting minutes

Date: 15/04/2025
Attendees: @rbeucher , @Aidan , @clairecarouge , @RachelLaw , @MartinDix , @tiloz , @inh599 , @pearseb , @dhb599 , @dougiesquire , @manodeep , @micael , @spencerwong , @paulleopardi , @Jhan

New spinup analysis

Ian showed first results:

  • Ocean temperature is falling due to the solar constant, need more time to reach equilibrium.
  • Salinity: New exe has change the salinity trend from slight decrease to slight increase. lprec value also changes the trend in the salinity as expected.
  • Surface energy and water balance just as good as before.
  • TOA balance too high at the moment because the ocean is still not at equilibrium. The land is also producing a -.3 imbalance which will stay.

Ian: Should the run with lprec=0 be continued knowing it is creating an increasing salinity? Probably having a non-zero value is ok because the model will never perfectly conserve water and this parameter compensates for that.
Tilo: Letā€™s continue both runs for another 100 years. No harm in letting it run for another week. Agrees that using a non-zero value for this parameter could be justified.

Pearse: The ocean primary production is too high now. To investigate and start running some tests to adjust the production. Question on where to find the configuration.
Spencer: Added links in the chat to the configuration and how to clone it with payu.

Ian: Should the previous spinup be continued?
Ian: Itā€™s not essential at this time. Tilo: happy to move on and continue with the current configuration.

Timing for starting the production runs

Claire: When would we need to freeze the model and the configurations so that we have enough time for the spin up to equilibrate before starting the production runs?

Tilo: The plan is still to start production runs from as early as possible in the next quarter. We want to run as many configurations as possible in parallel: PI-control, 1%, 4xCO2. From now on, changes should be fine tuning than big changes.

Ian: Over the next 2 months, we should check all the capabilities we will want to use are tested and assess (landuse change, emission drivenā€¦).

Rachel: We need to run other configurations to check the behaviour of the model under other conditions. It would be good to reassess the timeline considering we now know the runtime for the model.

Paul: it would be good to have dates forcing will be needed by so that itā€™s easier to plan the work.

Next meeting
22nd April.

33

Meeting minutes

Date: 22/04/2025
Attended: @rbeucher, @MartinDix, @tiloz, @RachelLaw, @dougiesquire, @spencerwong, @paulleopardi, @dave, @Jhan

Cable workflow confusion

Rachel: stash c with esm1.6 wanting to add some extra fields: soil respiration on tiles. But why were there missing variables? Martin: currently comes from different files (https://github.com/ACCESS-NRI/access-esm1.6-configs/blob/dev-preindustrial%2Bconcentrations/atmosphere/prefix.PRESM_A). Would be good to consolidate a workflow into a single file.

New cmip7 forcings

Rachel: Might want to try 4x CO2 and 1% runs. Martin: should be straightforward. Paul: wonā€™t be the official CMIP7 ones because we donā€™t have all the relevant files yet but can still give us an idea.

Chat update from Paul:

Only 7 out of 12 forcing datasets are currently available for CMIP7: Summary - input4MIPs Controlled Vocabularies (CVs) Even the testing phase only has 8 out of 12. The good news is that at least two forcings are not used by ESM1.6.

Mediterranean getting more saline like CM2?

Martin: Ian has highlighted some salinity issues, Mediterranean was getting more saline (ACCESS-ESM1.6 development - #183 by inh599) but CM2 was worse (~50 sos). Martin: suggested it remained a problem in CM2 but didnā€™t seem to create problems elsewhere so wasnā€™t fixed. If there was a known fix here it would be worth rolling in but perhaps not worth explicitly investigating. Arnold is not available to look this week but Tilo is having a look at a run he shared (not a totally clean comparison due to other changes).

Concern over Spack build using a branch to build from rather than a specific commit/tag. UM changes were rolled in when?

@Jhan: raised concern over Spack build using a branch to build from rather than a specific commit/tag?
@spencerwong: yes, that did occur but has since been updated to use specific tags/commits.

Note that the build system allows one to see the exact commit that was used for a PR. Example

image
image2059Ɨ946 87.5 KB

Chris: emphasized the importance of documenting experiments to mitigate this kind of confusion, especially those used for production runs in July.

Runlogs which are uploaded to the experiments repository also include a git history of the executables, restarts, and input files used during the experiments in the manifests directory.. This can be useful for finding what was used in a particular experiment, long after it has been run. For example, the git history for manifests/exe.yaml file for the first spinup experiment in the experiments repository shows that the same UM executable was used in all the years that have been uploaded to the repository.

Action items:

  • Please update this table and push relevant experiment information here ā€“ runlog needs to be set to True. Further details [0].
  • @dougiesquire to add details about how to find the file that tells you if/when the executable was changed. (exe manifest?)
  • @Jhan to check if he has missed pushing a repo, we looked at this page and he thought it looked like it was missing.
  • @pearseb (others?) needs to update the experiment table and push the updated runlog to this branch. Note this discussion for our suggestions of what to include in the table.
  • @spencerwong to put details here as to when the change from building from branches to tags/hashes occured:
    • The change to building from tags/hashes rather than branches occurred on February 10 for test PRs. This change was brought into the main branch on March 12.

Chat update from Paul:

From a branch you can work out a ā€œno later thanā€ date. This one is 22 January 2025
ACCESS-ESM1.6/spack.yaml at c214308600c3f545d48fc7076c7491eddbe1460f Ā· ACCESS-NRI/ACCESS-ESM1.6 Ā· GitHub
The last UM7 commit before 22 Jan was https://github.com/ACCESS-NRI/UM7/commit/1089e256dcec8ad3c900c294661b2cd5fef22565

land use change

Spencer: land use change ancillaries. Rachel: nothing to update at the moment.

cice5 updates

Dave: cice5 in place of cice4 is technically working in ESM1.6. Anton has been doing some tests, the results are not currently good enough to be integrated. Tried different initial condition strategies. Dave is still doing tests in esm1.5 but has other problems. Still needs input from Anton and Siobhan sea ice expertise to tune parameters. Anton is on leave this week.

Romain: no updates.

ā€“

[0]
At a past ESM 1.6 standup [1], we showed how weā€™ve come up with a workflow for storing peopleā€™s experiments for provenance. The idea is that this will support the spin up documentation provided here.

In short, weā€™re asking if you could please have a look at the instructions here:

https://github.com/ACCESS-Community-Hub/access-esm1.6-dev-experiments

Then push your esm1.6 experiment run folder to that repo with this branch naming convention: -<dev/test>--

Then add a link to the relevant experiment commits here:

https://forum.access-hive.org.au/t/esm1-6-spin-up-experiments/4219

Experiment Documentation

34

Meeting minutes

Date: 29/04/2025
Attendees: @tiloz , @clairecarouge , @MartinDix , @Jhan , @Aidan , @pearseb , @manodeep , @micael , @inh599 , @dougiesquire , @harshula , @anton , @paulleopardi , @RachelLaw , @alexnorton

Spinup update:

  • Jhan: latest run has crashed this weekend. Waiting to know why and make sure the interpretation of the error message is correct. Itā€™s the first crash in 350 years, showing a stable simulation.

    • Martin: It likely just needs a restart perturbation.
    • Aidan: No point trying to chase that down now, there is also less people available to help currently. No point of worrying too much about it unless it reoccurs. Payu will save the PBS information anyway.
    • Tilo: there was a conversion issue previously, but it was likely due to an NCI issue. It was manually fixed.

  • Ian: On the status of evaluation, the surface salinity seems to equilibrate now in the runs.

    • Jhan: it looks like the surface salinity climbs up again in the run that kept going. So we need to still keep an eye out for the surface salinity.

CICE5:

  • Anton: CICE5 running thanks to Dave. Ran for 50 years and it looks ok. Dave, Anton, Siobhan and Martin are to look at the results in more detail tomorrow, anyone is welcome to join.
  • Canā€™t use CICE4 restarts with CICE5. A short-lived hit is expected to the water balance when we switch over. Is there a water balance script available to check? Martin has a script for CM3, which could be adapted. Anton to figure it out since no script seems to be available for the ocean water balance.
  • Do we want to move to CMOR variable names in CICE5 and MOM? Seems a good idea but it would break user scripts. Manodeep suggests using aliases in the netcdf file so the data can be accessed using the old or the new names. Unsure if thatā€™s possible in netcdf (possible in hdf5) and that would not be supported by the models directly, it would need to be a post-processing step. Tilo: Itā€™s best to keep consistency in the output for the current spinup so itā€™s easier for evaluation. Only change for the next update to the spin-up.
  • Anton will still update the configuration so itā€™s ready to pick up when we want to.

Other discussions

  • Ian:

    • Reminder for Pearse and Jhan to update the spinup experiment page on the forum.
    • ILAMB workflow: need to meet with claire by the end of the week to get this going.

  • Pearse:

    • Ocean BGC update seems to have worked as expected. Fluxes look much better.
    • Needs help to update the diagnostics table to add some variables. Dougie to help.

  • Ian:

    • Thanks ACCESS-NRI for the work on fixing the output post-processing. It is now working very smoothly.

Actions:

1 Like
35

Meeting Minutes

Date: May 6, 2025
Chair and minutes: Chris Bull

Attendees:

Opening Remarks

  • Chris Bull opened the meeting; Claire is on leave, so he will be chairing the meetings for the next few weeks.

Discussion on Executable and Configuration Changes

  • Dougie Squire raised a point about changes requested by @Jhan regarding the executable and configuration for the spin-up.
  • Jhan is proposing some plant phenology types file changes. Would require an executable change but Dougie wanted to check in on the timing
  • Config changes. Pearse has added some things to the diag table (currently sitting on the head of the dev-preindustrial concentrations branch)
  • @inh599, the land people would like to add two new plant types (Australian-isation of land). Requires a build/config change together. And changes to a restart/ancillaries. Would suggest doing the adds at the same time. And more broadly would like to see a test containing all the changes that weā€™d like to be using for production.
  • Would be worth tagging the new version.
  • @dougiesquire advocated for a two phase approach with only a small number of release tags being used. Upcoming features/changes get accumulated on dev branch with spin off tests as needed. Once weā€™re happy we can create a tag and then that will get picked up by @Jhanā€™s control run.
  • @tiloz prefers small incremental changes

Next action:

  • create a new tag and then Dougie/Jhan can chat offline to get it being used in the spin up run. Will check in with @inh599 / @RachelLaw (discussed above/below) as to whether they should also include other land changes.

CMIP7 input project

The CMIP7 GHG forcings for ESM1.6 are defined here

Problem statement, @paulleopardi: Define mapping from land use forcing to ancillaries within ESM1.6. Needs scientific input to get the mapping. Alex Norton and Rachel are working on that. Greenhouse gases are simple to update (could be worth doing in the above updates?).
Next action:

  • Rachel wrote some Fortran and Alex has converted to python
  • Rachel suggests that they meet, Paul will follow up after this meeting with a DM.

Ocean update on CICE work.

Anton FYI: CICE5 update will need a payu update.
Anton: Discovered CICE5 configuration is accumulating a lot of water (Anton) have had offline meetings to assess the scientific quality.

Dave: Ocean dynamics are not currently very good currently. Southern Ocean circulation is currently too weak.

Salinity issue in the Mediterranean

Salinity in the Mediterranean slides from Martin (not working), Ian showed Figure from GitHub. Eastern Mediterranean going up linearly (western looks ok), trend is ~0.5/100years. See this Figure. Possible fixes? Ian: CM2 bug reimplementation / local land use changes.

Next action:

  • Martin, Ian, interested people could follow up with an offline meeting.

Other

@harshula: anyone have any Payu questions?

None raised.

36

Meeting Minutes

Date: May 13, 2025
Chair and minutes: Chris Bull

Attendees:

@cbull
Ian
Rachel
Tilo
Jhan
Ian
Jemma
Martin
Pearse
Alex
Anton
Harshula
Manodeep
Micael

General Updates:

  • Jhan reported issues with Gadi and /g/data4, affecting project runs. Noted recent NCI email update.

Pre-Industrial Greenhouse Gases:
- Martin Dix reported that the pre-industrial forcings (greenhouse gases, volcanic and aerosol forcings) are sorted and are almost identical to CMIP6, with minor differences. They look almost the same as cmip6 (greenhouse gas ones), but we are now using the cmip7 ones. Ian: land cover not included? Martin: relying on Rachel. Rachel: 3.1.2 is out ā€“ Alex is working from this. Martin: Ozone is still not released.

Related GH issues:

4xCO2 run:

  • Ian: In terms of where to start.. Wary of the salinity issue using a 4xCO2 forcing. Going as late as possible? i.e. as close to equilibrium as possible. Martin: are there any other major changes planned? Ian: yes, wanting to do a land cover changes. Martin: sounds like itā€™s best to wait.
  • Tilo/Rachel: suggests doing it asap as itā€™s a preliminary test that we can re-do when the other changes have been added. Martin: next action will set that up.

Land Cover Changes:

  • Ian and Rachel discussed progress on updating land cover types, particularly for Australian vegetation.

  • The team is working on parameter values and configuration changes. Requires code change (almost ready) and config change. Alex: been some progress but some work to do.

  • Ian: Still working Australian specific land cover types. Ian: Rhaegar has created scripts for CMORlite ACCESS-ESM can be used by iLamb but thinks Rhaegar still needs to tweak it. Chris: Is there a GitHub issue on this? No. Chris will follow up with with Rhaegar and Romain.

Ocean and BGC Updates:

  • Pearse will try and have a look at BGC output by Friday. Havenā€™t looked in the last two weeks but did have a look after last field table change. Tilo: air-Sea flux of CO2 was around zero (around last 100 years), was looking better than what we had in esm1.5.
  • Anton mentioned ongoing issues with freshwater accumulation in sea ice in cice5.
  • Chris Bull highlighted the salinity trend Mediterranean issue discussed last week, with no new updates on proposed fixes.

Software and Build Updates:

  • Harshula mentioned Dougieā€™s work on adding CMake support for MOM5 builds. For esm1.6 cmake build will ultimately be used. Various build things need to be adjusted. 2/4 scenarios are reproduced.
  • Manodeep: performance profiling. Both of the codes (preindustrial / amip) are spending a lot of time in mpi. cice is mostly waiting in 90% mpi (expected). Currently have a divide by zero runtime error in pi config when using oneapi. Currently debugging with valgrind. Stay tuned.

Next Meeting: next Tuesday.

37

Meeting Minutes

Date: 20 May 2025
Chair and Minutes: Chris Bull
Participants:
Chris Bull
Rachel
Anton
Ian
Pearse
Romain
Aidan
Dave Bi
Dougie
Harshula
Jemma Jeffree
Ian
Tilo
Micael
Manodeep
Lachlan
Martin

Large update discussion with multiple components/threads

  • Discussion on timing and bundling of changes. Suggestion: we start a forum table which everyone can edit will the following information: Priority,
    Target date for inclusion (afterthought: GitHub issue link). The following things could be considered:

    • Wombat changes
    • Forcing changes (Martin: aerosols will be ready in a day or two)
    • Australian PFTs
    • Canopy height
    • Root distribution
    • CICE5
    • Build system changes (see below)
    • cleaning up the vegetation distribution (so land use change is working)
    • code change radiation / bgc only experiments
      Rachel: once we have the suggested changes, a hackathon could be useful.

  • CMT Option Discussion:

    • Current run uses CMT=1; proposal to switch to CMT=0. Rachel: question of when to do this? Jhan already running for a while, could potentially drop the run with 1 and pick up from 0.
    • Ian: Concerns raised about discarding ~150 years of existing data as CMT=0 may lead to negative impacts on East Africa rainfall and Mediterranean salinity.
    • Ian and Tilo prefer to turn CMT=0 on at year 650 from the existing run. Rachel: would suggest looking closer at outputs before making a decision.

  • Other Land Updates:

    • Australian PFTs, canopy height, root distribution, and vegetation distribution updates.

Ocean & Biogeochemistry Updates

  • CICE 5 Experiment:

    • Dave Bi has had a look at Antonā€™s latest run in the mass imbalance looks like it has been fixed (Siobhan found an issue ā€“ one of the namelist options was incorrect update_ocn_f ā€“ iceā€™s view of the ocean salinity was not being updated). On Friday Anton started off from Jhanā€™s run, from year 685, years run ~40, will aim to get to 100. Will update the issue. Dave Bi is looking good: no salinity or mass imbalance drift.

  • Southern Ocean SST Bias:

    • Anton: question does esm1.6 have the southern ocean warm bias that is common in these kinds of models? Dave Bi: would expect it to be similar.

  • Mediterranean Salinity:

    • Anton has asked Ezhil/Chris to analyse transport between eastern and western basins, before and after code change.
    • Ian noted global surface salinity has levelled off but this does not necessarily imply local stabilization. A script has been shared that enables one to look at the more regional level.

4xCOā‚‚ Run

  • Ian: 4x C02 would be helpful for a few meetings in a few weeks.
  • Tilo: 4x C02 is already running, Martin has been having a look. /scratch/p66/txz599/access-esm/archive/test_4xCO2-expt-862aa61d

Build System & Technical Updates

  • MOM5 Build System:

    • Dougie: two upcoming answer changes in the pipeline: updated mom5 build system (OASIS MCT) and grid format. Hoping that answer changes will not lead to drastic scientific change. Change will be coming in a couple of weeks.
    • Testing required before integration.
    • Dougie will add to above table

  • Manodeepā€™s Update:

    • Investigating Pi config crash using Valgrind; work in progress.

Action Items & Next Steps

  • Create a table of all proposed changes (Chris to initiate).
  • Review and validate the CMT=0 run before switching.
  • Continue running and analyzing CICE5 and 4xCOā‚‚ experiments.
  • Monitor Mediterranean salinity and SST biases.
  • Prepare for potential hackathon or extended session to integrate changes.
  • Rachel and others will contribute to the checklist.

Afterthought! (not discussed) **

  • Next Meeting may be cancelled due to staff retreat. Will follow up with Rachel and others.
38

Meeting minutes

Date: 03 June 2025
Chair and Minutes: Claire Carouge
Participants:
Claire Carouge, Chris, Rachel, Anton, Jhan, Dave Bi, Romain, Pearse, Martin Dix, Harshula, Manodeep, Lachlan, Aidan, Micael, Alex, Dougie, Spencer, Ezhil

Actions from last meeting

Table of proposed changes

  • Chris has created the table.
  • Rachel has organised the information as ā€œscienceā€ changes that will change the results and ā€œtechnicalā€ changes that will not change the results.
  • Discussion on the science changes priorities at NESP meeting tomorrow. Anyone who wants to attend is welcome to contact Rachel for the meeting details.

CMT=0 option

  • It will be discussed in details at the NESP meeting.
  • Rachel found there isnā€™t much difference for climate simulations between the 3 options when used consistently.
  • A Met Office report supports this finding.

CICE5

  • Anton has completed 110-year run, aiming for 200 years.
  • Daveā€™s analysis shows:
    • Good strong overturning circulation in the Antarctic as well as stable ACC transport, close to observation. The ACC transport is much better than in CM2.
    • Sea-ice extent is comparable to ESM1.5 and 1.6 (with CICE4) and reasonable. Still showing too much ice in the northern hemisphere.
    • Better ENSO period.
    • All checks show the performance is as good as before or better. See GitHub issue for details.
    • Dave to add Jhanā€™s spinup results to his comparisons.

Mediterranean salinity issue

  • Ezhil is looking at the Sicily strait. Testing a simulation with a deepen strait.
  • Will test a wider strait afterwards if required.
  • Dougie noted we want to update the grid format for MOM, these changes need to be done in the new grid format. He will liaise with Ezhil.

4xCO2

  • 4xCO2 run is showing lower climate sensitivity at 3.4Ā°C (vs 3.9 in ESM1.5).
  • The current hypothesis is a link with the change in sea-ice in the simulation. Martin is still looking into it.
  • Rachel proposes to try a 1% simulation to compare.

Aerosol emissions for Fast-Track

  • Martin is running a test with the updated aerosol emissions.
  • They are pretty similar to CMIP6 and no large change is expected.

OneAPI dependency on CICE5

  • Manodeep to coordinate with Anton to set a ESM1.6-CIC5 test with OneAPI.

Data storage

  • Peter Dobrohotoff has identifies some space in /g/data/p73 for development runs.
  • Pearse asked for what data to keep from his ~900-year spinup.
  • Rachel to provide a list from a CMIP7 paper. Agreed on retaining annual averages only.
  • Discussed moving the data to CSIRO tape using existing scripts once it is cleaned up and in /g/data/p73/

ESM1.6-AMIP

  • Rachel wants to start an AMIP run initialised with a restart from the spinup.
  • Spencer says it should be possible to run. He will look into it.

Action items and next steps

  • Prioritisation of scientific changes at NESP project meeting.
  • Dave to add Jhanā€™s spin-up results to his CICE5 analysis.
  • Continue testing Mediterranean salinity fix.
  • Martin to continue investigating the climate sensitivity of ESM1.6.
  • Manodeep to start an ESM1.6-CIC5 test with OneAPI.
  • Rachel to share CMIP7 variable list with Pearse.
  • Pearse to proceed with data transfer.
  • Spencer to assist Rachel with AIMIP configuration.

Next meeting

  • next Tuesday (10 June)
39

Meeting minutes (10/06/2025)

Chair: @clairecarouge
Attendees: @clairecarouge, @cbull, @RachelLaw, @pearseb, @harshula, @Jhan, @lachlanswhyborn, @manodeep, @MartinDix, @micael, @rbeucher, @spencerwong, @tiloz, @jemmajeffree
Minutes: @cbull

@RachelLaw summary of the NESP meeting.

  • Spoke about: CMT option (0 or 1?), current run is 1 but 0 looks better for ENSO spectrum. Does seem to be sensitive to analysis time windowing period.
  • Related: how do we evaluate changes as we go? There is some confusion about best practice protocol for doing branch tests and how/when releases should be done. CICE5 was looking good but most recently there has been a salinity issue in the Arctic.

@Jhan: only manual changes have been Pearseā€™s field table (correct?) changes? Pearse clarified that the config (dev_preindustrial) has been updated to be consistent with this change. @RachelLaw suggests to @Jhan that the ESM1.6 experiments table be kept up to date.

@spencerwong has been looking at iceberg flux in CICE5. Looks like there is a mismatch with the landmask causing some fluxes to be erroneously inserted into land (maybe ~20%?)

@cbull. Gave an update on the Mediterranean work that @ezhilsabareesh8 has been doing, tests are promising but Ezhil is still doing some analysis.

@manodeep, oneapi progress. Still in progress.

@pearseb has started the discussed data transfer. Been moving monthly means for the atmosphere. Ocean: temperature, salinity, velocities. BGC tracers. Please get in touch with Pearse if youā€™d like anything else off scratch.

@lachlanswhyborn has been working on how CABLE is used. Linking it as a library, rather than including as source code in the UM7 repository. Ongoing.

@RachelLaw: anyone thinking about abstracts for the ACCESS-NRI community workshop? Currently, submissions close on the 15th June. @spencerwong and @MartinDix are currently thinking about a presentation on validating compiler changes.

40

Meeting minutes (17/06/2025)

Chair: @clairecarouge
Attendees: @anton @clairecarouge @paulleopardi @RachelLaw @inh599 @Jhan @dhb599 @dougiesquire @manodeep @micael @rbeucher @spencerwong @tiloz @alexnorton
Minutes: @cbull

Perspectives from UKMO meeting (@paulleopardi @inh599)

  • GC5-central looks like itā€™s mature now. ECS is approximately 3.6K.
  • Met Office will look to use UKESM1.3 for CMIP7 FASTTRACK. This is using GC3.1, which has an ECS of 5.6K and is going at ~4 years/day. (They use a relatively up to date UM, ~70 vertical levels with a lot more atmospheric chemistry.)
  • Met Office/UK universities will look to use UKESM2 (GC5-central) for CMIP7 Community MIPs.
  • Our capability of doing 20 years/day is something we can celebrate/consider an advantage. (Context CM2 is ~10 years/day)
  • [Another point @inh599 remembered after the meeting that is worth noting] The UK community realised (then identified) further issues with UKESM1.x when running historical and historical-amip tests (albeit with CMIP6 forcing). We maybe wise to set up/run such configuration/tests soon if only to provide a benchmark for later tests.

Spin-up runs update

@RachelLaw and @Jhan: we have a new June spin up run going, currently at 92 years. New format for phrenology file. Have decided to use CMT=1. The table has been updated. The relevant config is here and build here.

The question then, what is next? CICE5 or Australian PFTs perhaps. [@RachelLaw I think I missed a few other options here? ]

Coordinating development threads for next phase of control experiment

@cbull requested that folks keep the coordination table up to date. @RachelLaw this has been happening, see text Added to branch. @spencerwong noted a few things have been ā€œmergedā€ but that isnā€™t reflected, will up date.

Australian PFTs

GPP collapse has been fixed, due to using the wrong pft_params input file.
Updates simulation with PFTs 12 and 13 is still not reproducing the old simulation with same PFTs in 2 and 4. Traced back to the albedo in first time step: this is done with MOSES parameters and not CABLE. The differences then propagate.

@Jhan to change the parameters for MOSES and test.
@RachelLaw it raises the question whether there might be other MOSES parameters we are using and we are forgetting about.
@inh599 will look at the long term climate of the longer run to assess if itā€™s worth digging further.

CICE5 update

@anton: not a lot of activity recently. Spencer has run a CICE5 extension of the run that @anton was doing. Path to output will be shared/posted on GH issue such that @dhb599 and @sofarrell can have a look.

@RachelLaw query about odd salinity. @anton: odd salinity values were due to a Fortran bug, now fixed.

OneAPI status

@manodeep. PIconfig does now work with oneapi executables, the divide by zero workaround has slowed things down. Been using 2025.04.4 [correct number @manodeep?] but the new version doesnā€™t fix it either. Longer term, looking into lodging a bug report. Looking to coordinate with Anton on a longer CICE test.

Mediterranean salinity issue

@RachelLaw, any updates? @ezhilsabareesh8 is still looking into this, preliminary results are promising but @ezhilsabareesh8 is working on some more east vs west metrics.

Data move

@inh599: @pearseb has moved some data off scratch ( /scratch/p66/pjb581/access-esm/archive/pi_concentrations-expt-c55f7217/).
@manodeep: suggest creating a symlink to not break peopleā€™s workflows

Concern that early parts of the spin up are getting close to their expiration date. @clairecarouge would anyone like to follow this up in more detail? @tiloz offered to follow up.