Encountering error while running simulations

COSIMA Ocean
, , ,
1

Hi Spencer,

I’m not sure if this issue is related to the Payu version, but I’m encountering the following error while running one of my simulations located in /home/561/hs2549/lgm_topo:

/g/data/vk83/apps/payu/1.1.5/lib/python3.10/site-packages/payu/runlog.py:110: UserWarning: Error occured when attempting to commit runlog

payu: Model exited with error code 134; aborting.

Could you please take a look and help me figure out what might be causing this?

Thanks a lot,
Himadri

2

Hi Himadri,
I had a quick look at the run dir. I think this commit runlog thing is just a warning, it’s not the fatal error that crashed the run.
If you check the access.err file, the first error message (I search for the first instance of forrtl) shows the following information:

forrtl: error (75): floating point exception
Image              PC                Routine            Line        Source
mom5xx             0000000001D17464  Unknown               Unknown  Unknown
libpthread-2.28.s  000014A41223AD10  Unknown               Unknown  Unknown
mom5xx             0000000000E13E84  ocean_tempsalt_mo         725  ocean_tempsalt.F90
mom5xx             0000000000E0BF68  ocean_tempsalt_mo         939  ocean_tempsalt.F90
mom5xx             0000000000B70BD0  ocean_tracer_mod_        2466  ocean_tracer.F90
mom5xx             00000000005313C8  ocean_model_mod_m        1865  ocean_model.F90
mom5xx             0000000000F52425  MAIN__                    433  ocean_solo.F90
mom5xx             000000000040F162  Unknown               Unknown  Unknown
libc-2.28.so       000014A411E8C7E5  __libc_start_main     Unknown  Unknown
mom5xx             000000000040F06E  Unknown               Unknown  Unknown

So it looks like the ocean temperature-salinity is going unstable somewhere.

I also checked the file work/atmosphere/atm.fort6.pe0 and searched for Timestep and it looks like the atmosphere crashed after 72 timesteps. This suggests that the atmosphere initiated ok, but then something went unstable.

3

Given that 72 timesteps of the atmosphere is 1.5 days, I would suggest to change the run length of this simulation to 1 day, so you can check the ‘before’ and ‘after’ ocean_temp_salt.res.nc. This might give some clue as to which region of the ocean is starting to go off the rails?

4

Thanks, David. I’ll try that.

5

Hi @HIMADRI_SAINI,
Just so you know we have moved your question to it’s own topic and tagged it with help, it may be an ocean issue instead of payu.
Please reply back if you found the issue and you can mark this as resolved.
Or @ezhilsabareesh8 maybe you can have a look?

7

Hi @HIMADRI_SAINI Just checking in to see if David’s suggestion helped resolve the issue with your simulation. Let me know if you need further assistance in this topic.

8

Hi @ezhilsabareesh8. Thanks for checking in. David’s suggestions were helpful, but the issue isn’t completely resolved yet. It might have something to do with the simulation setup rather than the model configuration. I’m testing a few things and will revisit this. Thanks.