Atmosphere Post-processing Not Triggered in ESM1.6 experiment

Earth System
, ,
1

Hi @WillemH,

I saw your comment in the ACCESS ESM1.6 experiment segfault topic and thought the issue you raised was worth discussing separately, so I’ve opened a new topic for you here.

I believe @spencerwong has already looked into this and opened a GitHub issue to track it: payu #888.

2

Just following up, I’ve noticed that since using the tool to modify the restarts, the atmosphere variables are no longer being post-processed after each year finishes its integration. Specifically, the archive/atmosphere/netCDF directory with its output is not produced and instead aiihca files can be found in the atmosphere folder. I’m not sure what I could have done to break the post processing (in that I haven’t changed anything in terms of the experiment setup), does anyone have any thoughts on this and, equally as importantly, how can I post-process these data manually?

3

Hi Willem,
Having a quick poke at it… I see that your postscript (bottom of your config.yaml) is calling the UM_conversion_job.sh which itself is calling esm1p5_convert_nc

In the more recently released ESM1.6 (released at beginning of September), there is a different post-processing tool, which looks like this:

module use /g/data/vk83/modules
module load model-processing/1.0.0_0

# Convert UM atmosphere fields files to netCDF.
# By default UM atmosphere fields files are deleted after conversion to save space.
# Remove --delete-ff command line option to retain original files for testing purposes
# Source code for the um2nc command is available at https://github.com/ACCESS-NRI/um2nc-standalone/
um2nc driver esm1p6 $PAYU_CURRENT_OUTPUT_DIR --delete-ff  --one-nc-per-stash-variable

… so I suspect something might have changed “under the hood” of the old post-processing tool (which was set up for esm1.5), and this is entirely coincidental to your restart file changes.

I don’t know why it breaks per se… but if you check out the September release of the PI control there would be updated “post-processing” tools that should work retrospectively on your archived outputs.

I imagine @spencerwong might be better placed to know why it’s breaking now when it wasn’t before.

NOTE: the new tool as written splits the atmosphere outputs into single variable files. That’s very different to the old way it used to happen. Thus I would recommend doing some non-destructive testing (try without the --delete-ff flag) before applying it wholesale to your outputs.

4

Hi @WillemH, as mentioned in the linked issue there appears to be a problem with the post-processing when runspersub>1. This leads to multiple postprocessing jobs targeting the same output directory and the others being missed. We’re working on a fix now and for the mean time recommend setting runspersub: 1 for any remaining simulations.

Unfortunately it looks like 1-3 of atmosphere outputs in each of the directories output303, output333, output351, output381, output401, and output431 have been corrupted, likely due to multiple postprocessing jobs trying to work on the files at the same time. Some of the converted atmosphere files don’t to contain the full amount of data:

16M	aiihca.pa-043203_mon.nc
4.0K	aiihca.pa-043204_mon.nc
16M	aiihca.pa-043205_mon.nc

which could be caused by multiple post-processing jobs working on the same files.

As the original UM files aren’t available for these years and as there’s a large gap between available restarts, recovering these particular outputs would require rerunning this section of the experiment. Would you have resources available to do this? If not, there will likely be resources available through the working groups.

Otherwise if those particular outputs aren’t required, you can manually rerun the postprocessing command for the remaining directories with the command:

qsub -P<project> -v PAYU_CURRENT_OUTPUT_DIR=<path-to-outputXYZ> -lstorage=gdata/vk83+scratch/<project> scripts/post-processing/NetCDF-conversion/UM_conversion_job.sh

Apologies for the inconvenience caused by this issue @WillemH!

5

Thanks Spencer. I didn’t even notice there was a runspersub flag. @WillemH if you’re wondering about how to run many years with one submission, the usual way is to just submit the job like this:
payu run -n <yrs>
So that each submission only runs for ~1 hour, but payu will resubmit itself to do however many years you set using the -n flag.

6

Indeed. The advantage of using runspersub is you have less queue time. You still use -n to say how many time you want the model to run, it just runs multiple jobs per queue submission.

Obviously I wouldn’t recommend that until we have a fix available!

7

Hi all,

Thanks for your input, much appreciated! I think the best thing for me will be to re-run this part of the experiment, particularly as I cannot post-process all the data I’ve already copied to the HPC here at PIK without considerable extra effort. I will run a short test with runspersub = 1 to ensure the old post-processing script for v1.5 still works, as I don’t feel like changing file structure half way through my experiment. Will keep you all posted if there’s any further issue.