ESM1.5 payu: Model exited issue

Earth System Earth System Model
, , ,
1

Hi Spencer,

I’m facing a similar issue like Himadri, with the error “payu: Model exited with error code 1; aborting.” at restart184.

I’m not sure if this is a leap year problem, but I tried using the warm-start.sh script and the run still stops after ~50 minutes (same as before), with the same error.

I also tried to delete the previous output and restart files, and perturbing the atmosphere restart file (the model runs a bit longer to ~1 hour 14 mins), but same error still occurs. I suspect maybe it is more related to the error with “payu: Model exited with error code 1; aborting.” that I need to delete some files that exited before.

I’m wondering if you have any thoughts about it.

Thank you,
Yanxuan

2

Hi @YanxuanD
Thanks for your post!

The error “payu: Model exited with error code 1; aborting” can be a difficult one to find the cause of. My understanding is that whenever any of ESM’s submodels (e.g. atmosphere, ocean, ice) crash during a simulation, that crash will be picked up by Payu, which then reports the "payu: Model exited with error code 1; aborting” error to the user. As a result there could be a large range of different causes.

Would you be happy to share the path to your experiment’s control directory? I can take a look over some of the logs to see if anything stands out.

It’s interesting that applying the perturbation changed the amount of time the model ran for. This could mean that the model was crashing due to an instability, which the perturbation helped avoid, only for the model to run into another issue later on. I’m not completely sure though and so it would be great to see what some of the logs are saying!

3

I’ve just moved this to a separate topic in so that specific advice can be added here.

4

Hi @spencerwong, thank you so much for the reply!

Of course, it would be great if you could take a look at the logs. I’m running at my home directory at /home/563/yd3797/49ka_shutdown, the archive is at /g/data/ob22/yd3797/49ka_shutdown/.

Please let me know if you have troubles accessing them.

Thanks!

5

I just looked and in access.err and found this error:

FATAL from PE  123: ==>Error from ocean_tracer_util_mod: The maximum salinity is outside allowable range

So the ocean model has crashed because the salinity is too high.

Some steps you could take to diagnose this:

  1. Look in accèss.out in the work directory and in previous ouputXXX directories for messages that report maximum salinity and the location, e.g.
 The maximum S is     5.472107809042E+01 psu at (i,j,k) = ( 314, 199,  2),  (lon,lat,dpt) = (   33.5000,   31.4997,   15.0000 m)
  1. Check ocean salinity diagnostics to see if there is an obvious problem with high salinities using the values reported above

Note the maximum salinity is s_max:

and this is set to 55 in your ocean/input.nml:

 &ocean_tempsalt_nml
      temperature_variable='conservative_temp'
      pottemp_2nd_iteration=.true.
      t_min=-5.0
      t_max = 55.0
      s_min = -1.0
      s_max = 55.0
      t_min_limit =-2.0
      t_max_limit =32.0
      s_min_limit =0.0
      s_max_limit =42.0
/

Given the model defaults to s_max=70.0 it is presumably safe to increase the value in ocean/input.nml to 60.0 and keep an eye on it to make sure it doesn’t just keep drifting upwards indicating a systemic problem in your configuration.

6

Hi Aidan,

Thanks for the help! Now I understand why this error occurs.

For this experiment, I keep adding freshwater to continue the AMOC shutdown at 49ka for as long as possible. I have more than 200 years of shutdown now, and this is probably why the ocean salinity exceeded the threshold.

Thank you again for identifying the problem!

1 Like
7

Hi Yanxuan,
Note that such a high salinity is very odd and should not happen even for a multi-centennial North Atlantic meltwater addition. Please check where it is occurring (I cannot see the j and k location from the post). We have to make sure this is not linked to the change in land-sea mask. Also, you did not use any salinity compensation in the experiment, right? i.e. Is your global salinity decreasing or staying constant?
Laurie

8

Hi Laurie,

No I didn’t use any salinity compensation, and I think this may be linked to the change in land-sea mask at 49ka. This figure below is the sea surface salinity during the last 50 years of shutdown. The mean SSS value in the Mediterranean Sea is close to 55 psu.

download-1

9

OK, interesting. The Med is indeed supposed to be much drier during H events. Then maybe the best is to follow Aidan’s suggestion and increase s_max to 60 in ocean/input.nml.
@HIMADRI_SAINI , did we reduce the Gibraltar Strait much?

10

No, it’s quite similar to PI.

11

Can I just check, did this simulation include the new river runoff for 49ka topography that Himadri was running with? If the answer is no, it could be somehow related to new topography no longer lining up with where the river outflows are meant to be. If the answer is yes, and we do in fact have updated 49ka rivers, then it might be a bit more subtle to fix this.

12

It is with the updated 49ka rivers, David.

13

Thanks Himadri. Yes in this case it might be worth increasing the salinity limit as a starting point. Longer term if the salinity is still accumulating, you might need to widen the choke points of the Mediterranean to make sure that its flushing out high salinities better. By this I mean making some small tweaks to the topography. That might be something to tackle in Q4 though!

It might be worth doing a quick comparison to the pre-industrial bathymetry to gauge whether this configuration is more blocked at certain places than the pre-industrial one.

14

Thanks, David. That’s a good point.

15

I think it is because the figure size of this map is not showing the correct scale and a bit stretched vertically, that’s why the Gibraltar Strait looks a bit magnified.