WOMBAT crashing in Miocene ACCESS-ESM1.5 simulation

Earth System
, , , ,
1

Hi all,
Reposting here an email exchange I’ve been having with @spencerwong.
I am trying to run WOMBAT in my ACCESS-ESM1.5 simulations of the Miocene. Previously I have spent most of my effort on getting the physical model running. I switched off WOMBAT a while ago in order to deal with other issues. Now resurrecting my efforts to re-introduce WOMBAT.

Currently the model is crashing very early in the simulation (6 atmos timesteps). The run is located at:
/home/157/dkh157/ACCESS/esm_b21_wom_c3

The error message is coming from L1170 of csiro_bgc.F90.

It is an error (75): floating point exception.

Looking at the code, it seems likely that the model is somehow receiving a negative value of sc_o2, which would trigger a sqrt(negative). Spencer helpfully pointed out that this could occur when temperature exceeds 107 degC, which could occur if it was accidentally receiving temperature in K.

I don’t know how or where in the model this might be occurring. As far as I know, the physical circulation is ok and has run now for 1000 years.

Any help would be most appreciated.

2

Weird things can happen to physical fields if the model goes unstable.

Is there any possibility the model has gone unstable, which they sometimes do, and this just happens to be the first thing that crashes?

3

Hi Aidan,
The mystery here is that the model runs quite happily if I switch off the WOMBAT modules. I.e. The physical circulation seems to be very stable.
I am still investigating if I’ve done something strange to my boundary conditions that feed into WOMBAT. I have suspected initially that the error could be caused by my FICE or XKW input fields, but if that were happening, I would expect it to crash at Line 1167… indeed the similarity between 1167 and 1170 suggests the sc_o2 is the culprit.

1 Like
4

Another thing I don’t understand is which input files is WOMBAT actually reading in. In the “old” configuration, WOMBAT had input forcing files for

XKW
fice
dust
pressure

which all followed the land-sea mask, (hence I adapted them to Miocene equivalents). In the ACCESS-NRI 2024 release, these inputs seem to have been reduced to only:

dust
pressure

So, I’m guessing XKW and fice are no longer required, if using the release version?

5

XKW and fice were removed from the ACCESS-NRI-released configuration because they were not actually being used. From ocean/field_table:

"namelists","ocean_mod","csiro_bgc/*global*"

gasx_from_file = .false.
ice_file4gasx = .false.

Are you wanting to provide those files in your configuration?

2 Likes
6

So this is six time steps since you turned on WOMBAT?

7

Yes… 6 atmos timesteps.

8

Thanks Dougie, no didn’t want to especially, just learning how the inputs for WOMBAT work.

9

@dkhutch my blind guess is that this is occurring because the temperature has gone sufficiently high to cause the Schmidt number of O2 to go negative. The formulation used to calculate the Schmidt number is only really valid up to 30C or something. It is common to limit the temperature to a valid range when performing this calculation, but this is not done in the ESM1.5-version WOMBAT (it is done in newer versions of WOMBAT).

If my guess is correct, I’m not sure what the immediate workaround is. I can put together a version of ESM1.5 that includes the limiting for you to test?

1 Like
10

Thanks @dougiesquire, that sounds like a great idea.
Another idea that comes to mind is can we somehow output the t_prog field that seems to be triggering the error? Is t_prog coming from the coupler? Or from the ocean prognostic fields themselves?

11

I can also do a brute force test e.g. set the ocean temperature restart file to 10 degC everywhere and see if the model survives for longer before WOMBAT crashes.

12

t_prog contains the prognostic tracers. You’ll want to look at the ocean surface temperature.

13

Hi @dougiesquire,
Ok, so the brute force test seems to work. I reset the temperature and salinity to uniform values of 10degC and 34.7 psu. The model is currently running and has completed > 600 atmos timesteps without crashing.
That would suggest it is indeed getting too hot in the ocean and causing some sort of blow-up of sc_o2.

1 Like
14

Great. I’m currently building ACCESS-ESM1.5 with limiting included. I’ll let you know when it’s ready and how to use it. It shouldn’t be long.

It would be great if you could test it out. If it fixes your issue I think we can do an official release with the fix.

1 Like
15

This is now ready to test. To use the version of ACCESS-ESM1.5 that includes the fix, replace the modules section in your config.yaml with:

modules:
  use:
      - /g/data/vk83/prerelease/modules
  load:
      - access-esm1p5/pr29-1

Your config.yaml is probably set up to disallow using different executables than what are specified in the manifest. You’ll need to turn this off by changing:

manifest:
  reproduce:
-    exe: True
+    exe: False

(Note you can turn this back on after updating the manifest if you like. I.e. change to False, run payu setup, change back to True)

Let us know how you go!

3 Likes
16

Thanks @dougiesquire. Currently testing the new pre-release version you’ve specified here. (And reverted back to my hot TS restart file.)
It is currently running without error and has so far completed > 500 atmos timesteps.
I will wait for it to complete one year, and if so, will mark your post as the solution.
Many thanks!!

2 Likes
17

“WOMBAT crashing in Miocene” . . . I couldn’t help myself. :smiling_face_with_sunglasses:

wombat crash
wombat crash1024×1024 163 KB

3 Likes