Netcdf conversion on existing ESM1.5 configurations

Earth System
,
1

The ACCESS-NRI releases of ESM1.5 include netcdf conversion as a post-script to the payu run step:

If you have existing configurations you are still running, you can add this script to the configuration folder and add the postscript line (linked above) to the config.yaml line. (This might require the use of payu in the vk83 released folder).

If you have existing output which needs converting to netcdf, then you can trigger the conversion manually. Make a copy of UM_conversion_job.sh, and add

#PBS -P ${PROJECT}
#PBS -l storage = gdata/vk83+gdata/${PROJECT}

Change this line

esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff

to not delete the original files (--delete-ff) and include the path to the output.

esm1p5_convert_nc .../archive/outputxyz

If you have multiple output folders, you can do something like:

for f in .../archive/output*[0-9]; do
    esm1p5_convert_nc $f
done

and make sure to make the wall-time longer =, e.g:

#PBS -l walltime=12:00:00

Corrections welcome (@hrsdawson & @spencerwong)

3 Likes
2

Thanks Anton, that’s really helpful for retro-fitting to existing runs.

1 Like
3

Looks good to me. For walltime reference, converting one year of ESM output (monthly files) took ~15-20 minutes for me.

1 Like
4

Hi Hannah,
This is useful performance data for the um2nc redevelopment project. How many files were you processing & do you have any details of the total volume of data being converted?

5

This was 24 files (12 aiihca.pa* files, and 12 aiihca.pe* files) which had this size originally:

Screenshot 2024-09-23 at 4.27.46 PM
Screenshot 2024-09-23 at 4.27.46 PM864×632 222 KB

And this is the converted and compressed size:

Screenshot 2024-09-23 at 4.29.55 PM
Screenshot 2024-09-23 at 4.29.55 PM978×427 110 KB

Note: the script didn’t convert the aiihca.pg* files, but these only seem to contain a couple of tracer flux fields that I’m not interested in, so I didn’t mind - others might?

1 Like
6

Thanks @hrsdawson.

In case you’re not familiar, du -shc will print the sizes of a directory or file glob in human readable format and provide a total at the end, which is quite useful. Just FYI.

1 Like
7

Did not know, thanks @Aidan.

1 Like