# Netcdf conversion on existing ESM1.5 configurations

**URL:** <https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706>\
**Category:** Earth System\
**Tags:** postprocessing, netcdf\
**Created:** [23 September 2024 01:58 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706 "2024-09-23T01:58:43Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\
**Post date:** [23 September 2024 01:58 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/1 "2024-09-23T01:58:43Z")

</div>

The ACCESS-NRI releases of ESM1.5 include netcdf conversion as a post-script to the `payu run` step:

> <https://github.com/ACCESS-NRI/access-esm1.5-configs/blob/4c2543ccf76dabaeb9346c2f72c18f716f3b4559/config.yaml#L162>

If you have **existing configurations you are still running** , you can add this script to the configuration folder and add the `postscript` line (linked above) to the `config.yaml` line. (This might require the use of payu in the vk83 released folder).

If you have **existing output which needs converting** to netcdf, then you can trigger the conversion manually. Make a copy of [UM\_conversion\_job.sh](https://github.com/ACCESS-NRI/access-esm1.5-configs/blob/dev-preindustrial%2Bconcentrations/scripts/NetCDF-conversion/UM_conversion_job.sh), and add

```bash
#PBS -P ${PROJECT}
#PBS -l storage = gdata/vk83+gdata/${PROJECT}

```

Change this line

```bash
esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff

```

to not delete the original files (`--delete-ff`) and include the path to the output.

```bash
esm1p5_convert_nc .../archive/outputxyz

```

If you have **multiple output folders** , you can do something like:

```bash
for f in .../archive/output*[0-9]; do
    esm1p5_convert_nc $f
done

```

and make sure to make the wall-time longer =, e.g:

```bash
#PBS -l walltime=12:00:00

```

Corrections welcome (@hrsdawson & @spencerwong)

---

<div class="post-metadata">

**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\
**Post date:** [23 September 2024 03:05 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/2 "2024-09-23T03:05:50Z")

</div>

Thanks Anton, that’s really helpful for retro-fitting to existing runs.

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [23 September 2024 03:11 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/3 "2024-09-23T03:11:17Z")

</div>

Looks good to me. For walltime reference, converting one year of ESM output (monthly files) took ~15-20 minutes for me.

---

<div class="post-metadata">

**Author:** ![bend](https://avatars.discourse-cdn.com/v4/letter/b/f04885/32.png) [@bend](https://forum.access-hive.org.au/u/bend)\
**Post date:** [23 September 2024 06:04 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/4 "2024-09-23T06:04:29Z")

</div>

Hi Hannah,  
This is useful performance data for the `um2nc` redevelopment project. How many files were you processing & do you have any details of the total volume of data being converted?

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [23 September 2024 06:33 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/5 "2024-09-23T06:33:34Z")

</div>

This was 24 files (12 `aiihca.pa*` files, and 12 `aiihca.pe*` files) which had this size originally:

 ![Screenshot 2024-09-23 at 4.27.46 PM](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/1/1451067a3460d8476733c8ff71786291d2ddf66a.png)

And this is the converted and compressed size:

 ![Screenshot 2024-09-23 at 4.29.55 PM](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/5/51e3b0351e8054b85aeb8c09ad74ac6732a96824.png)

Note: the script didn’t convert the `aiihca.pg*` files, but these only seem to contain a couple of tracer flux fields that I’m not interested in, so I didn’t mind - others might?

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [24 September 2024 02:36 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/6 "2024-09-24T02:36:43Z")

</div>

Thanks @hrsdawson.

In case you’re not familiar, `du -shc` will print the sizes of a directory or file glob in human readable format and provide a total at the end, which is quite useful. Just FYI.

---

<div class="post-metadata">

**Author:** ![hrsdawson](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/hrsdawson/32/218_2.png) [@hrsdawson](https://forum.access-hive.org.au/u/hrsdawson)\
**Post date:** [24 September 2024 03:31 UTC](https://forum.access-hive.org.au/t/netcdf-conversion-on-existing-esm1-5-configurations/3706/7 "2024-09-24T03:31:01Z")

</div>

Did not know, thanks @Aidan.