Hi @WillemH, as mentioned in the linked issue there appears to be a problem with the post-processing when runspersub>1. This leads to multiple postprocessing jobs targeting the same output directory and the others being missed. We’re working on a fix now and for the mean time recommend setting runspersub: 1 for any remaining simulations.
Unfortunately it looks like 1-3 of atmosphere outputs in each of the directories output303, output333, output351, output381, output401, and output431 have been corrupted, likely due to multiple postprocessing jobs trying to work on the files at the same time. Some of the converted atmosphere files don’t to contain the full amount of data:
16M aiihca.pa-043203_mon.nc
4.0K aiihca.pa-043204_mon.nc
16M aiihca.pa-043205_mon.nc
which could be caused by multiple post-processing jobs working on the same files.
As the original UM files aren’t available for these years and as there’s a large gap between available restarts, recovering these particular outputs would require rerunning this section of the experiment. Would you have resources available to do this? If not, there will likely be resources available through the working groups.
Otherwise if those particular outputs aren’t required, you can manually rerun the postprocessing command for the remaining directories with the command:
qsub -P<project> -v PAYU_CURRENT_OUTPUT_DIR=<path-to-outputXYZ> -lstorage=gdata/vk83+scratch/<project> scripts/post-processing/NetCDF-conversion/UM_conversion_job.sh
Apologies for the inconvenience caused by this issue @WillemH!