CSIRO - ACCESS-NRI standup minutes

CMIP7 CMIP7 development
21

Meeting Minutes
Date: February 18, 2025
Attendees: Chris Bull, Anton, Pearse Buchanan, Claire Carouge, Ian, Tilo Ziehn, Aidan Heerdegen, Dave Bi, Martin Dix, Manodeep Sinha, Micael Oliveira, Tammas, Dougie, @paulleopardi
Minutes: @cbull

Agenda:

  • Project Board Review
  • Timelines for Work
  • Model Validation and Spin-Up Process
  • Ancillary Files for Production Runs

Discussion Points:
Project Board Review:

Chris Bull asked if there were any issues or blockers regarding the project board.
Pearse Buchanan noted a few ocean things were labelled in progress even though they are complete, he will update. Claire Carouge emphasized the importance of closing pull requests to reflect completed tasks.

Claire Carouge suggested (next action for everyone) reviewing the number of “Todo” issues and setting priorities or removing from project board if not within scope.

Timelines for Work:

Chris Bull inquired about the timelines for work and the version of the model to be used for production runs in July. Tilo Ziehn explained the current status of the ocean spin-up and the integration of land model updates.

The goal is to use the long simulation runs by Pearse Buchanan for the spin-up (currently at year 543). There is not a target number of years (rather equilibrated metrics) but currently the model gets around ~25 years per day.

Chris Bull highlighted the importance of coordinating work tasks to meet the July deadline for fast-track production runs.

Tilo Ziehn discussed the plan to integrate model updates and run the spin-up continuously with CABLE being added when available (new cable version currently being tested in ESM1.5). The aim is to have a stable model configuration by the end of June. (Timing of the integration of CICE changes was not discussed.) This is not the best practice ‘continuous’ spinup that the CMIP protocol requires to be publishable.

Side point: Ian asked Tilo about recent CABLE changes and Ian offered to have a look at them, Tilo Ziehn said the run is available: /scratch/p66/txz599/archive/PI-restart/history/atm which is the current test run with the new vegetation fractions. Still raw binary output, but can extract variables in netcdf as required.

NB: Manodeep/software transformation team are able to do their best work when the model configuration is finalised as soon as possible.

Model Validation and Spin-Up Process:

Aidan Heerdegen proposed automating model validation tests to reduce manual inspection. Pearse and Aidan discussed what that could look like and that it’s quite flexible.
Dave mentioned the availability of automatic tools for model performance evaluation. Chris suggested Dave is free to share more about that at future stand up sessions.

Ancillary Files for Production Runs:

Paul raised a question about the ancillary files needed for the production runs:

which CMIP7 forcings for ESM1.6 need to be ready for the spinup in July. Pre-industrial, historical, other?

As a list of priorities, Tilo and Ian agreed it is: picontrol, 4x1pct, historical. Tilo Ziehn emphasized the importance of having the necessary ancillary files ready for the July production runs.

Action Items:

  • Review and prioritize outstanding Todo issues on the project board (All).
  • Continue the ocean spin-up and integrate land model updates (Tilo Ziehn, Pearse Buchanan).
  • Consider development and implementation of automated model validation tests (Aidan Heerdegen, Dave, Pearse).
  • Discuss timeline for CICE changes in ESM1.6 at next week’s standup.

Call for other business regarding ocean and land teams:
No items.

Next Meeting: 25/02/2025. Claire will facilitate.

22

Thanks! Please add my name to the attendees, since I am mentioned in the minutes.

23

Thanks @paulleopardi, now added (note you can edit as these are wiki posts). Unfortunately AI seems to get this wrong every time – so need to remember.

24

Thanks!

25

Meeting Minutes
Date: February 25, 2025
Attendees: @anton , @clairecarouge , @inh599 , @Jhan , @rbeucher , @Aidan , @arnoldsu , @ben , @harshula , @manodeep , @MartinDix , @micael , @paulleopardi , @pearseb , @spencerwong , @tiloz , @cbull

CICE5 implementation timeline concerns
Claire and Tilo discussed the timeline for CICE 5 implementation, noting that it might not be ready in time for the production runs. They agreed getting more visibility of the work soon is critical. Getting the code in GitHub by the end of April at the latest is essential for any hope to integrate CICE5 in ESM1.6.
Tilo and Chris and others concerned will discuss with Dave and Shioban when Dave comes back from leave.

Problem with the netcdf conversion for UM files
Ian and Jhan noticed the netcdf conversion is sometimes failing. This causes issues for quick evaluation of runs with standard scripts. Aidan and Spencer are aware of the issue and looking into it.

Cmorisation of outputs
Romain is now attending the stand-ups. He is working with Arnold on preparing the cmorisation for CMIP7.
He asked for clarity on the raw output format that will be produced. He will also document all the calculations done on variables for review by the group.

New executable for long runs
Ian said he and Jhan are consolidating several parallel developments in a single source code. These changes also include changes in the configuration and in payu.
In about a week, we are aiming to replace the executable for the long running simulation with this new source code as well as bringing in some changes on the ocean side. The timeline depends on how many sanity checks we want to do beforehand.
Manodeep mentioned if we change the executable it might be good to run on Sapphire Rapids and if possible to compile with the OneAPI compiler. Moving to Sapphire Rapids show a 20% reduction in CPU time and a reduction in cost.
Spencer mentioned adding more CPUs for the atmosphere didn’t show the same improvements as for ESM1.5. This is because in ESM1.6 the atmosphere is waiting on the ocean and not the other way around.

Forcings creation
Paul now has access to scripts used for CMIP6. He needs additional explanations from Tilo or Martin to understand how the scripts work and how to modify them for CMIP7.

Actions

  • Tilo to organise a meeting for CICE5 when Dave comes back.
  • Aidan and Spencer to investigate the collation and conversion issues
  • Ian and Jhan to consolidate the land developments into a single source code and configuration.

Next meeting
March 4th, 2025. Chris to facilitate.

1 Like
26

Meeting minutes
Date: March 4th 2025
@Jhan, @cbull, @harshula, @manodeep, @RachelLaw, @MartinDix, @tiloz, @micael, @anton, @spencerwong, @paulleopardi, @inh599, @clairecarouge, @ben, @Aidan, @dougiesquire

Agenda Items

  • Chris Bull no pre-existing agenda items.

Communication Update

  • Claire Carouge highlighted the move of our ESM1.6 communications from a hidden message to a public topic.
  • Chris Bull clarified the renaming of the thread and encouraged attendees to watch the thread for notifications (click bell down the bottom).

Project Board Review

  • Chris Bull reviewed the ESM 1.6 project board and invited attendees to mention any blockers or issues.
  • No issues / blockers were mentioned.

Experiment Documentation

  • Chris Bull emphasized the importance of documenting experiments, especially those used for production runs in July. This process has begun already, thanks @spencerwong and @pearseb
  • Spencer Wong discussed the creation of a GitHub repository for experiment details and provided instructions for pushing experiment branches.
  • Specifically this branch naming convention <GH-user-name>-<dev/test>-<YYYYMMDD>-<counter> and the use of GH issues for any descriptors. Please see README for further instructions.
  • @pearseb not here today but please add experiment for current spin up experiments to this repo’.
  • Note this is a different GH organisation so you’ll need to get in touch with @clairecarouge / @spencerwong to get access.

General Matters

  • Claire Carouge raised an issue about building and running ESM 1.6, mentioning a runtime error related to generic tracers.
  • Dougie Squire and Harshula discussed potential issues with the build system and configurations and the need for updates on both.
  • Dougie Squire wanted to know which config branch to focus on, this one

Data Management Plan

  • Ian raised a concern about data storage and the potential for data purging on scratch storage.
  • Rachel Law and Tilo Ziehn discussed the need for a data management plan, especially for spin-up and production runs. Only a subset is probably practical to keep on disk the rest can go on tape.
  • Paul Leopardi requested a copy of Chloe’s original data plan for reference.
  • No one is currently assigned to write a Data Management Plan (afterthought: @cbull will ask @ClareCR about next steps?)
  • @Aidan payu has in-built sync support. If we’re in danger of losing important information it should be turned on

Compiler and MPI Updates

  • Harshula expressed the need to use a newer compiler and OpenMPI version for future test runs. Namely openmpi: 4.1.5 / ifort: 2021.10.0. See PR. When is the best time to roll out these changes?
  • No one followed up at standup. Chris would follow up offline.

Action Items

  • Chris Bull to follow up on the sequencing of changes related to the new compiler and MPI version.
  • Rachel Law to provide Paul Leopardi with a copy of Chloe’s data plan.
  • Dougie Squire to review and update ESM1.6 configuration to ensure compatibility.

Closing Remarks

  • Dougie Squire Looks like I won’t have time to raise my hand. I want to update the ocean grid, which will also entail updating the ice grid and remapping weights: link
  • Chris Bull Out of time today, please send agenda items in advance such that we can plan for time as appropriate
  • Chris Bull thanked attendees and closed the meeting. @clairecarouge is hosting next week.
27

Meeting minutes
Date: March 11th, 2025

@Jhan , @cbull , @clairecarouge , @harshula , @manodeep , @RachelLaw , @MartinDix , @tiloz , @arnoldsu , @spencerwong , @pearseb , @paulleopardi , @rbeucher , @inh599 , @ben , @Aidan , @dougiesquire , @tammasloughran

OneAPI new executable

  • Manodeep discussed the move to OneAPI and whether to apply the change in one go or incrementally.
  • Martin expressed developers don’t need access to intermediate steps and switching to OneAPI in one go would be fine.
  • Aidan and Harshula mentioned the pre-release infrastructure allows for several executables to be created within a single PR. We should take advantage of this so that, during the development phase of the OneAPI support, changes to the compiler can be done incrementally and tested.
  • Dougie mentioned changes to the configuration might be needed in step with changes to the compiler. Currently, ACCESS-OM2 executable using OneAPI crashes the model. More investigation is needed to understand where it comes from.
  • Manodeep asks if there are tools to compare results of 2 configurations. Dougie said there are and will discuss this further with Manodeep.

Software update for ESM1.6 build

  • Harshula mentioned Dougie has a software update PR for the ESM1.6 build that we should finish first and use as a base for the OneAPI work.
  • Ian: there is a blocker here as there might not be a UM version everyone would be happy with. The latest version shows questionable results in a test run. A previous version is problematic as well because it does not have restart reproducibility. This needs a scientific discussion at CSIRO to decide how to fix the issue. [INH edit: All previous ESM1.6 versions do not have run-to-run reproducibility, ESM1.6 will never have restart reproducibility. It is the run-to-run reproducibility that is necessary.]
  • Tilo agreed we need to solve the question of the executable for the spin up quickly and it will need a meeting of the parties involved.
  • Rachel said we might be able to create an updated ESM1.6 build that is compatible with the configuration to enable the compiler work to move forward. And work on solving the issue with the UM version for the spin up separately.

Sharing experiments with the community

  • Chris said he has a meeting with Pearse, Spencer and Dougie to explain the workflow on how to share ESM1.6 experiments on the community repository. Anyone interested is welcome to attend.

Ocean grid update

  • Dougie asked whether we should update the ocean grid to avoid supporting a decade-old grid. This could simplify the support for the cmorisation. However, it would require changes to a number of tools and the remapping weights between the atmosphere and ocean grids.
  • Rachel had a concern about the land mask. Dougie and Martin assured the land mask would not change sufficiently (rounding errors) to be a concern.
  • Paul said the change will have an impact on the preparation of the CMIP7 forcings for the ocean.
  • Martin and Aidan agreed it would be good to update the ocean and sea-ice grids. Using an old grid format will make it much more difficult for Model Evaluation to support.

Action Items

  • Ian and Tilo to initiate meeting on solving the issues with the latest UM7 version with interested parties
  • Dougie to continue work on updated ESM1.6 build if possible
  • Dougie to initiate work on updating the ocean and sea ice grids.

Next Meeting

Tuesday, March 18th. @clairecarouge is hosting next week (@cbull is an apology).

1 Like
28

Meeting minutes
Date: March 18th, 2025

@paulleopardi , @MartinDix , @rbeucher , @manodeep , @clairecarouge , @tiloz , @RachelLaw , @anton , @spencerwong , @tammasloughran , @dougiesquire , @inh599 , @harshula , @micael , @Aidan , @dhb599 , @pearseb , @arnoldsu , @Jhan

CMIP7 forcings
We need a written description of the processing applied to the forcings See what was produced for CMIP6: https://www.publish.csiro.au/ES/ES21031 (section 2.3)

Paul to open a topic on the forum to gather input on the treatment of the forcings for CMIP6 and what needs to be done for CMIP7.

We will introduce the CMIP7-based forcings when they are available to use in ESM1.6

New spinup configuration
Temperature drift
New tests haven’t identified the cause of the drift. For the moment, tests seem to point to the calculation of canopy%epot but it’s still preliminary.
Need to review the whole setup of the simulation, Aidan and Spencer are happy to help with learning how to use payu effectively.

Outputs Collation
To solve recurring issues in the file collation after the simulation, it was decided to write the ocean outputs in a single file directly. This comes at the cost of 5 min in walltime. There is a bug with some MOM5 diagnostics if output to a single file, this wasn’t deemed major.

Sapphire Rapids
Changes are done and ready for people to use. People would need to rebase their development branches, contact Spencer. The change brings a 20min improvement in walltime.

PI restart with new veg. distribution
Some of the values in the file are not internally consistent. Problems disappear after a few years of running. Will pick up a restart from later in the simulation to remove these effects.

Question from Lachlan about the new for values for the previous year, Rachel replied we don’t need the previous year for the spinup phase.

Data management plan:
Peter Dobrohotoff looking into it but not attending the standups. It might be best to discuss it at the leadership meetings.
We also need to look into the output profiles we are using: it seems we are saving more data than we need. Nobody earmarked to look into it yet.

Non-determinism with changes in processor decomposition
A problem was identified that took about 6 months to appear.
Solution: have a cron test that runs every week to pick up when the model becomes non-deterministic. Aidan says the machinery is already here to make this happen (see issue).
Generally, Dougie says we need to run repro tests more routinely. Repro testing is installed in the access-esm1.6-config repository, people need to create pull requests in the configuration repository to trigger these.

Switch to OneAPI
Manodeep says configurations with OneAPI are ready, question about the timing for the switch of compiler.
Martin and Tilo: next spinup setting to run with ifort first. It will be good to be able to run a long simulation (~100 years) with both ifort and OneAPI to assess the effect on the climate before a definitive switch.

29

Meeting minutes
Date: March 25th, 2025

@tiloz , @clairecarouge, @RachelLaw , @manodeep , @MartinDix , @micael , @spencerwong , @pearseb , @ben , @harshula , @tammasloughran , @Aidan , @anton , @Jhan , @inh599, @dougiesquire (late)

Martin: Run with the new, reduced solar constant using Jhan’s latest config. Shows a cool down as expected but nothing dramatic so far.

Harshula: Openmpi 4.1.5 update - no effects on results and negligeable effect on performance.

Jhan: New couple of simulations using latest version of UM7, one with epot change and one without the epot change. Get the same results for both tests, still looking for why there is no difference. Ian: the 2 new test runs show good results compared to Pearse long run with very minor changes to the average ocean temperature.

Forcings: volcanic changes will be very small. Looking at the aerosols next.

ILAMB update: Romain gave an update at the CMIP7 Leadership meeting last Friday. He expects this to be ready in 2-3 weeks. Claire to contact Rhaegar to keep track of this activity.

Compiler update: When trying a simulation from a build with an updated ifort compiler, Harshula and Dougie had similar performance but different results. The next step is unclear. Question from Claire whether it is useful to investigate further considering the plans to change to OneAPI.

Walltime: Jhan has 20 years/day using Sapphire Rapids. Ocean outputs to one file per variable which would increase the walltime slightly. But Jhan noticed problems with the atmosphere conversion. Spencer to have a look.

Pearse: Update to the ocean BGC parameters to come. Long run at year 900.

Dougie: no progress on the new grid.

Dave: CICE5 still under way. Issue with initialisation, trying to resolve this.

Ian: the land group is trying to add some Australian plants in the model. New feature to come.

Rachel: Farewell to Tammas, finishing his post-doc this week

Action:

  • Make sure your build PR on ACCESS-ESM1.6 repository are rebased onto the latest main branch. Talk to @harshula if you need help or more information.
30

Meeting Minutes

Date: 01/04/2025
Attendees: @paulleopardi, @clairecarouge @micael @Jhan @tiloz @inh599 @spencerwong @manodeep @cbull @MartinDix @rbeucher @pearseb @dhb599 @Aidan @dougiesquire @anton @ben
Minutes: @cbull

Martin has a new run CMIP7 solar constant

@MartinDix’s new run has 0.3-0.4 degree cooling in the global mean. Will post some new Figures on the thread.

Paul’s update on CMIP7 forcing files

@paulleopardi has been working on the CMIP7 pre-industrial forcing files for use in ESM1.6. See issues for current challenges and wiki for current understanding of ESM1.5 cmip6 workflow.

Discussion on around which version of the land updates to merge into the current spin-up run

  • Ian raised concerns about the combination of land code changes merging into @pearseb existing pre-industrial run. At the moment they ( @inh599 @tiloz ) have some unexplained results and a lack of confidence that the build / config system is actually running the code they expect. Considering parking the current line of work due to time constraints and relative importance.
  • Claire Carouge suggested involving someone more familiar with the build system to fix issues rather than parking them.
  • Dougie Squire and Jhan emphasized the importance of tracing builds and understanding the changes.
  • Dougie/Claire/Chris: up to you as to how much time you have to persue it but it really helps if the build/config is on GitHub (example). Sounds like an issue worth getting to the bottom of. Next action: @Jhan to share the relevant builds on forum or GitHub. @Aidan: feel free to ask for reviews of payu/config use and also helpful if you can put plots on GitHub issues.

Further Land Team Updates

  • Ian mentioned progress in updating the ESM with new Australian vegetation PFTs (e.g. spinifex, eucalyptus), though it may be more complex than initially thought.
  • Claire Carouge gave an update on ILAMB. Rhaegar is now able to run ILAMB on ESM1.6 outputs. He will put together a workflow so anyone can use ILAMB with ESM1.6 outputs.

Software transformation team and MED team

  • Manodeep Sinha reported on the rerun of tests due to a non-deterministic bug in the UM which is being looked into in AMIP, confirming deterministic results across cascade lake/sapphire rapids. Looking to merge into esm.
  • Romain been using lite version of MOPPER to create CMOR’isation of output. Will post on the forum about next steps.

Ocean Team Updates

  • Pearse Buchanan has been continuing the long runs. Branched off main run at about ~1000 years because there was a slight change, now no longer considering burial of matter through sediments. No longer an open boundary at the bottom for the carbon cycle – seems unsurprisingly that’s bringing things closer to equilibrium. There will be another ecosystem component update soon (Pearse/Dougie hoping to merge wombat sinking PR today – slight answer change). Chris requested that the forum documentation and config repo get updated accordingly.
  • Dave has created a CICE5 branch for ESM1.6, put all of his updates of the modules there. Dave has had some trouble compiling. Anton has tried compiling and it looked like it built successfully (still need to check that the executable exists) – will look at it more closely today.
31

Meeting minutes

Date: 08/04/2025
Attendees: Romain, Claire, Harshula, Martin, Tilo, Michael, Manodeep, Jhan, Dougie, Dave, Aidan, Ian, Lachlan, Spencer, Pearse, Paul

Cascade Lake non-reproducibility
Manodeep: Configs now run on SapphireRapids (~20% faster). PI runs non-reproducible on CascadeLake cores when changing atmosphere layout. With the same ocean layout but different atmosphere layout. AMIP runs with any atmosphere layout are deterministic.

Micael and Martin agree it is not worth chasing further but important to document.

New WOMBAT-lite
@dougiesquire and @pearseb have released a new version of WOMBAT-lite on main. Can there be a new ESM1.6 build release using that update? It is almost identical to what Pearse has used in the spin up run. @dougiesquire cmake build stuff is still not in main, so could either release from dev tag or try to get that in. @Aidan the latter would be better. @harshula to organise doing this asap.
It has also implications for the new spin up configuration we want to put in place. The ocean restart will need some manual change in the deep ocean for the ocean BGC to use the new WOMBAT-lite. (@pearseb to do).
When do we want to start the new spin up? @tiloz does not want to wait too much as there are more changes coming. It might be good to start before Easter so we can analyse and fix some problems before it runs through Easter time.

Decision: we shouldn’t wait too long to the new WOMBAT-lite. If it can come in today or about, it can go in the new spin up. Otherwise, it will get integrated later on.

Spin ups and tests

@tiloz says Pearse’s current spin up can still run in parallel to the new spin up so Pearse can test changes in the ocean, sea ice and BGC. Same approach to be used if we have new changes focused on the land later on.

Salinity decrease
@dhb599 Salinity decrease, have tests been done with a parameter change to solve this? @pearseb Dave’s suggestion seems to solve the decreasing salinity. @inh599 says we may want to undo that because new land changes that are coming in are slightly increasing the salinity. The parameter is a tunable parameter so can be modified depending on what an experiment produces.

CICE5 technically working
@dhb599 says CICE5 is now working technically in ESM1.5. Haven’t checked results yet. @anton and Siobhan need to analyse it.

32

Meeting minutes

Date: 15/04/2025
Attendees: @rbeucher , @Aidan , @clairecarouge , @RachelLaw , @MartinDix , @tiloz , @inh599 , @pearseb , @dhb599 , @dougiesquire , @manodeep , @micael , @spencerwong , @paulleopardi , @Jhan

New spinup analysis

Ian showed first results:

  • Ocean temperature is falling due to the solar constant, need more time to reach equilibrium.
  • Salinity: New exe has change the salinity trend from slight decrease to slight increase. lprec value also changes the trend in the salinity as expected.
  • Surface energy and water balance just as good as before.
  • TOA balance too high at the moment because the ocean is still not at equilibrium. The land is also producing a -.3 imbalance which will stay.

Ian: Should the run with lprec=0 be continued knowing it is creating an increasing salinity? Probably having a non-zero value is ok because the model will never perfectly conserve water and this parameter compensates for that.
Tilo: Let’s continue both runs for another 100 years. No harm in letting it run for another week. Agrees that using a non-zero value for this parameter could be justified.

Pearse: The ocean primary production is too high now. To investigate and start running some tests to adjust the production. Question on where to find the configuration.
Spencer: Added links in the chat to the configuration and how to clone it with payu.

Ian: Should the previous spinup be continued?
Ian: It’s not essential at this time. Tilo: happy to move on and continue with the current configuration.

Timing for starting the production runs

Claire: When would we need to freeze the model and the configurations so that we have enough time for the spin up to equilibrate before starting the production runs?

Tilo: The plan is still to start production runs from as early as possible in the next quarter. We want to run as many configurations as possible in parallel: PI-control, 1%, 4xCO2. From now on, changes should be fine tuning than big changes.

Ian: Over the next 2 months, we should check all the capabilities we will want to use are tested and assess (landuse change, emission driven…).

Rachel: We need to run other configurations to check the behaviour of the model under other conditions. It would be good to reassess the timeline considering we now know the runtime for the model.

Paul: it would be good to have dates forcing will be needed by so that it’s easier to plan the work.

Next meeting
22nd April.