ACCESS-ESM 1.6 job not ending

General
, ,
1

Hi all,

I’ve just begun using ACCESS ESM 1.6 and have run into an issue on my first 5 year test. The model appears to have finished, and the ocean output has already been combined (though not the ice or atmosphere diagnostics), but the pbs job won’t end despite the last of the file changes being almost 24 hours ago.

Has anyone else had this happen? If I kill the job now, which post-processing tasks remain? Thanks.

2

Hi @WillemH, I’ll try to find someone within access-nri to answer your query about the post processing part. The hanging might be more of a question for NCI however. Do you have access to the job logs to indicate where it’s hanging at?

3

Hi @edoyango , thanks for your reply. .out file ends here:

Total IO Timings: FILE_CLOSE
Calls: 189
Total time: 0.069741 secs                Average time: 0.000369 secs
Calling redsea_gulfbay_hmix_s at runtime = day=1825, sec=0, ticks=0

There are some warnings in the .err file, but nothing that would explain a hanging job (I’ll contact NCI helpdesk for this then).

4

Ok, just to add, the job just exceeded its walltime and was killed. The error file shows the job was here at the time it was killed:

Image              PC                Routine            Line        Source             
libpthread-2.28.s  000014650FF37990  Unknown               Unknown  Unknown
libc.so.6          000014650FA530A8  nanosleep             Unknown  Unknown
libc.so.6          000014650FA80968  usleep                Unknown  Unknown
libmpi.so.40.40.7  000014650CECC541  ompi_mpi_finalize     Unknown  Unknown
libmpi_mpifh_Inte  0000146510C8A97A  MPI_Finalize_f08      Unknown  Unknown
um_hg3.exe         0000000000EFC56E  oasis_terminate           677  mod_oasis_method.F90
um_hg3.exe         0000000000414054  um_shell                 4018  um_shell.f90
um_hg3.exe         000000000041112A  flumemain                  29  flumeMain.f90
um_hg3.exe         000000000041108D  Unknown               Unknown  Unknown
libc.so.6          000014650F985865  __libc_start_main     Unknown  Unknown
um_hg3.exe         0000000000410FAE  Unknown               Unknown  Unknown
5

Hi @willem,

ESM1.6 now writes the ocean output from a single processor, which means there’s no longer a post-processing task for the ocean. However the atmosphere netCDF post-processing remains and won’t have run yet if the payu job failed.

Based on the logs, it failed during the model run stage – are the outputs you mentioned still located under the work directory? If so, payu won’t yet have started moving data into the archive directory, and manually running the atmosphere post processing would be quite difficult.

I’d recommend retrying the failing run, but wanted to quickly check how you’ve set up your 5 year test experiment. Have you directly cloned one of the ESM1.6 configurations and run it with payu run -n 5, or have there been additional customisations?

6

Hey @spencerwong ,

Thanks for the info. Yes, the output is still in work , hence thinking it had crashed/ hung during post-processing (I’ve also never known a model not to terminate a compute job when it crashes, which also confused me) - thanks for clearing this up.

As for the set up, I’m using the most recent version of the pi control + emissions branch and modified some options in order to allow an idealised freshwater hosing for the ocean, in addition to building new diagnostic tables for the ocean. Additionally, I modified the config.yaml file to change the run length to 5 years and ran a payu run . I should note that this was the second test I ran, but the first also failed due to a bus error (though as that’s hardware related, I didn’t think was worth mentioning).

7

Hi @WillemH, thanks for adding that extra information about the setup. I’ve reproduced the issue with the job hanging by swapping to a 5 year run length.

It’s recommended to only run ESM1.5 and 1.6 using one year run segments and to run long simulations using the payu run -n command, as longer run times can lead to issues. In this case, it looks like the UM isn’t properly accounting for the impact of leap years in multi-year runs, leading to the atmosphere model terminating early and leaving the ocean and sea ice models hanging.

When using 1 year run segments, it’s still possible to run 5 years within a single PBS job via the runspersub option in the config.yaml. E.g, if you change

calendar:
    runtime:
        years: 1
        months: 0
        days: 0

-runspersub: 1
+runspersub: 5

and increase the the walltime appropriately, then running payu run -n 5 will do 5 consecutive 1 year simulations in a single job. Some more details on this option are available in the Hive Docs here.

Let me know if this resolves the issue with the model hanging

9

Hi @spencerwong ,

Thanks so much for testing this! I’ll give this a go and report back if the issue persists. One more quick question as it’s not entirely clear to me: if I submit successive payu run commands in the same experiment directory, subsequent runs are automatically treated as restarts, yes?

10

Hi @WillemH, yes that’s correct! A successive payu run from the same control directory will automatically pick up the newest restart in the archive directory to continue the simulation

11

Hi @WillemH, just wanted to check back in to see if the issue with the model stalling has been resolved or if you are still running into problems?

12

Hi @spencerwong , I’ve since managed to integrate 62 years without further issues, so I imagine the problem is solved. Thanks again for your help.