@Jhan @tiloz Is the plan to stop the current June-Spinup or run the two in parallel for a while?
We are not restricted by compute at the moment so we could keep the June spinup running until we are satisfied that CICE5 is working as expected in the new spinup.
I’ve just noticed we should update the coordination table as well when changes have been rolled in. I’ve added this to the checklist
Update coordination table on ACCESS-Hive forum.
Minutes from today’s standup, please correct/amend.
@inh599 @Jhan @MartinDix @alexnorton please update the Australian PFTs bit
@MartinDix @anton @spencerwong @Jhan @cbull @clairecarouge @tiloz @RachelLaw @AndyHoggANU @dhb599 @sofarrell
@AndyHoggANU asked me to copy today’s results across to the hive - I wasn’t quite sure where would be best.
Anyway - below is the progress with the spin up with purple being the AugustSpinup (up to year 152 as of the morning of 18/8/2025). Our ocean cooling rate in the latest section appears to have increased in magnitude - but salinity has stabilised. I’m not sure if this is solely a (transitory) response to CICE5 going in or indicative of something more problematic. Note the trend numbers are from the JuneSpinup (in green).
1 Like
Minutes from today’s stand up, pleas correct/amend.
After discussing with @tiloz , @RachelLaw and @inh599 , we agreed that, considering the current issues in the August spin-up, running the spin-up isn’t a priority right now. This is especially true as there are indications we may want to restart from an earlier date.
This means we don’t need a replacement for Jhan while he is on leave for now. We’ll see what the current tests tell us and reassess as needed.
@tiloz also potentially volunteered to run longer runs of potential candidate configurations if we need to.
Further food for thought regarding the ocean trends in ESM1.6 …
Figures shared here show some changes in ocean temperature in the new August spinup with respect to the June spinup.
There is little difference in the globally averaged SST between the two spinups (top left, black is June, red August).
However, this obscures regional differences; in particular, cooling around Antarctica and warming in the North Atlantic (bottom left).
There are corresponding increases and decreases in average sea ice cover (of the order of 5%) in the south and north, respectively.
As already noted, the average ocean temperature is cooling quicker in the August spinup (top right).
When checking the change in zonally averaged temperature sections (bottom right), this cooling is predominantly in upper 2000 m of the ocean and there is actually some warming in the deeper ocean.
This distribution corresponds to a decrease in Antarctic Bottom Water formation by ~10% in the August spinup corresponding to the switch to CICE5.
IMHO, there is less certainty about long term ocean temperature trends associated with changes in circulation/overturning than when changing insolation/albedo.
Note, differences to the June spinup are to 30 years of this spinup before branching to the August version.
3 Likes
Thanks @inh599 and @matthew.chamberlain for sharing some temperature drift insights (1 and 2).
@MartinDix has opened a GitHub issue on this. So we can keep the discussion confined, can you please copy your content across to the issue? One of the ACCESS-NRI people can then remove the posts above.
I had a few runs going over the weekend and had a few crashes.
I had one case of the ocean/wombat error message which I’ve added to floating invalid crash in Wombat in August spinup · Issue #182.
I had another JobFS exceeded error. Seems to happen at the end of the year when it’s trying to copy to the archive directory. Stops when it has tried to use ~10 times more JobFS (~800Mb) than is used in other cases (8.32 Mb). See /g/data/p66/rml599/amip-test/amip-exp6/archive/pbs_logs (with *.keep being a copy of the failed year).
When looking at why some of my jobs had failed, I happened to notice that the atm.fort6.pe207 file was much larger than the equivalent files for other .pe numbers. This appeared to be the same for all cases that I had running and looked like it might be due to output related to a river-routing error (@inh599, @spencerwong). Now my runs have finished, I don’t have the files but this directory (at least while the model is currently running) seems to be showing the same behaviour.
/scratch/p66/jxs599/access-esm/work/AugustSpinUp-Jhan-dev-20250808-1-d1b0b669/atmosphere
And one failure to convert to netCDF: /g/data/p66/rml599/amip-test/amip-exp7/archive/output009 - so I’ll run a payu collate.
@spencerwong I think this may be the consequence of one of the water fixes (this is certainly the rivers processor). In effect we’re getting a lot of
”!! rc is not positive”
error messages. I can’t remember exactly but I think we set rc=0 on inland basin points.
Minutes from today’s stand up, pleas correct/amend.
I have updated the coordinating table for the planned changes. Feel free to amend/correct
Minutes from today’s stand up, please correct/amend
Hi everyone, I’ve added some results comparing the climate in a 150 year run using oneAPI against the AugustSpinup here.
Discussing with @MartinDix, there don’t seem to be any clear problems, however the scale of the variability does make differences difficult to detect.
Minutes from today’s stand up, please correct/amend
Minutes from today’s stand up, likely a few details I missed/got incorrectly, so please correct/amend
An update on last week before tomorrow’s standup:
- Model build release that includes all the outstanding priority 0 items from this table (except for the forcings). These items are now moved to the “COMPLETED” section of the table. The release model version is:
access-esm1p6/2025.09.002 dev-preindustrial+concentrationsconfiguration updated with new ancillaries, restart and forcings.- Two candidates spin-up are set up and running. They correspond to the cases in this table. (@MartinDix I’ve added the branch names to the table, please check I’ve got the correspondence right.). Issue for the configurations
- First look at results: Global ocean temperature drift · Issue #188 · ACCESS-NRI/access-esm1.6-configs · GitHub
As @Jhan reminded me, I forgot the link to the experiments table in my update. This list the information about the experiments including where to find the outputs.