# How to use cumf or mule-cumf with ACCESS ESM1.5 UM output?

**URL:** <https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794>\
**Category:** Earth System Model\
**Tags:** reproducibility, access-esm-release\
**Created:** [7 February 2024 23:54 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794 "2024-02-07T23:54:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulleopardi](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paulleopardi/32/92_2.png) [@paulleopardi](https://forum.access-hive.org.au/u/paulleopardi)\
**Post date:** [7 February 2024 23:54 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/1 "2024-02-07T23:54:48Z")

</div>

In testing the ESM1.5 pre-industrial run, I noticed that there is a script that builds a version of the `cumf` executable that matches UM vn7.3. The script is  
`branches/dev/Share/VN7.3_local_changes/Install_scripts/UmBuildSmllExecs` within the `/g/data/access/access-svn/UM_Admin` repository.

I think that if we need to build this vn7.3 `cumf` and related executables we may need a private [https://github.com/ACCESS-NRI/UM\_Admin](https://github.com/ACCESS-NRI/UM_Admin) or [https://github.com/ACCESS-NRI/UM\_Admin\_v7](https://github.com/ACCESS-NRI/UM_Admin_v7) repository.

The alternative would be to get an existing build of `mule-cumf` working correctly for the UM vn7.3 output of ESM1.5.

Before I start anything in this direction, I need to know how current users of ESM1.5 are using `cumf` or `mule-cumf` and where they obtain the executable that they use. I do know that `mule-cumf` is working for others and I want to understand why it isn’t working for my username `pcl851`.

See [Create framework for extracting, verifying and updating model reproducibility and performance · Issue #1 · ACCESS-NRI/reproducibility · GitHub](https://github.com/ACCESS-NRI/reproducibility/issues/1)

See also [Porting CSIRO/UMUI ACCESS-ESM1.5 ksh run script to payu](https://forum.access-hive.org.au/t/porting-csiro-umui-access-esm1-5-ksh-run-script-to-payu/1611) and [https://code.metoffice.gov.uk/doc/um/mule\_utils/2019.01.1/um\_utils/cumf.html](https://code.metoffice.gov.uk/doc/um/mule_utils/2019.01.1/um_utils/cumf.html)

---

<div class="post-metadata">

**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\
**Post date:** [9 February 2024 00:09 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/2 "2024-02-09T00:09:01Z")

</div>

mule-cumf should work fine for old versions, what issue are you having?

---

<div class="post-metadata">

**Author:** ![paulleopardi](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paulleopardi/32/92_2.png) [@paulleopardi](https://forum.access-hive.org.au/u/paulleopardi)\
**Post date:** [9 February 2024 02:27 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/3 "2024-02-09T02:27:07Z")

</div>

As per [Create framework for extracting, verifying and updating model reproducibility and performance · Issue #1 · ACCESS-NRI/reproducibility · GitHub](https://github.com/ACCESS-NRI/reproducibility/issues/1)  
I see

```auto
[pcl851@gadi-login-05 access-esm]$ pwd
/scratch/tm70/pcl851/access-esm
[pcl851@gadi-login-05 access-esm]$ module use /g/data/hh5/public/modules/
[pcl851@gadi-login-05 access-esm]$ module load conda/analysis3-23.07
[pcl851@gadi-login-05 access-esm]$ module use /g/data/access/modules
[pcl851@gadi-login-05 access-esm]$ module load um
[pcl851@gadi-login-05 access-esm]$ mule-cumf --stashmaster 7.3 archive.orig/access-esm/restart000/atmosphere/restart_dump.astart archive.build-gadi.1/access-esm/restart000/atmosphere/restart_dump.astart 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py (version 2022.07.1)
um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/ __init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/ __init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)

/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.orig/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup) : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.build-gadi.1/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup) : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/mule-cumf", line 10, in <module>
    sys.exit(_main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 1385, in _main
    comparison = UMFileComparison(um_files[0], um_files[1])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 728, in __init__
    diff_field = difference_op([field_1, field_2])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py", line 952, in __call__
    new_field = self.new_field(source, *args, **kwargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 293, in new_field
    data1 = fields[0].get_data()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py", line 730, in get_data
    data = self._data_provider._data_array()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ff.py", line 193, in _data_array
    data = np.fromstring(data_bytes, dtype, count=count)
ValueError: string is smaller than requested size

```

Are the warnings and the `ValueError` to be expected, or do I have a misconfiguration?

---

<div class="post-metadata">

**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\
**Post date:** [9 February 2024 02:42 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/4 "2024-02-09T02:42:27Z")

</div>

They’re not expected, but they indicate odd properties of the files rather than something wrong with mule-cumf.

Three fields have an axis in the wrong order, and four fields claim to be timeseries (lbcode 31320), which is unexpected in a restart file. Finally it wasn’t able to read the data from at least one field.

Is one of these files somewhere where I can take a look at it?

---

<div class="post-metadata">

**Author:** ![paulleopardi](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paulleopardi/32/92_2.png) [@paulleopardi](https://forum.access-hive.org.au/u/paulleopardi)\
**Post date:** [9 February 2024 03:10 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/5 "2024-02-09T03:10:17Z")

</div>

I tried again with two different files that you should have access to:

```auto
[pcl851@gadi-login-06 access-esm]$ mule-cumf --stashmaster 7.3 /g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart /g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/atmosphere/restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py (version 2022.07.1)
um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/ __init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/ __init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)

/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: /g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup) : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4935,4937,6676,6715)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: /g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup) : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4935,4937,6676,6715)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/mule-cumf", line 10, in <module>
    sys.exit(_main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 1385, in _main
    comparison = UMFileComparison(um_files[0], um_files[1])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 728, in __init__
    diff_field = difference_op([field_1, field_2])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py", line 952, in __call__
    new_field = self.new_field(source, *args, **kwargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 293, in new_field
    data1 = fields[0].get_data()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py", line 730, in get_data
    data = self._data_provider._data_array()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ff.py", line 193, in _data_array
    data = np.fromstring(data_bytes, dtype, count=count)
ValueError: string is smaller than requested size

```

---

<div class="post-metadata">

**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\
**Post date:** [9 February 2024 05:39 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/6 "2024-02-09T05:39:32Z")

</div>

There’s a few corrupt fields. You may have more luck running the comparison on the model output files, presumably the corrupt data is ignored by the model.

You can remove the corrupt fields and get cumf to work with

```python
mf = mule.load_umfile('/g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart')

out = mf.copy()

for f in mf.fields:
    if f.lbuser4 not in [155,156,3100,3101,33001,33002]:
        out.fields.append(f)

out.validate = lambda *args, **kwargs: True
out.to_file("/scratch/hc46/saw562/access-esm.astart")

```

> **mule-cumf test**
>
> ```auto
> $ mule-cumf /scratch/hc46/saw562/access-esm.astart /scratch/hc46/saw562/access-esm.astart
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> * (CUMF-II) Module Information *
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> 
> mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ __init__.py (version 2022.07.1)
> um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/ __init__.py (version 2022.07.1)
> um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/ __init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)
> 
> /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning:
> File: /scratch/hc46/saw562/access-esm.astart
> Field validation failures:
> Fields (1114,1115,1116)
> Field grid latitudes inconsistent (STASH grid: 23)
> File : 145 points from -90.0, spacing 1.25
> Field (Expected): 180 points from -89.5, spacing 1.25
> Field (Lookup) : 180 points from 89.5, spacing -1.0
> warnings.warn(msg)
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> * CUMF-II Comparison Report *
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> 
> File 1: /scratch/hc46/saw562/access-esm.astart
> File 2: /scratch/hc46/saw562/access-esm.astart
> Files compare
> * 0 differences in fixed_length_header (with 7 ignored indices)
> * 0 field differences, of which 0 are in data
> 
> Compared 6554/6554 fields, with 6554 matches
> 
> ```

---

<div class="post-metadata">

**Author:** ![paulleopardi](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paulleopardi/32/92_2.png) [@paulleopardi](https://forum.access-hive.org.au/u/paulleopardi)\
**Post date:** [11 February 2024 23:57 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/7 "2024-02-11T23:57:22Z")

</div>

Thanks, Scott. I will try to run the model again using a restart file with the corrupted fields removed, to make sure that these fields are not needed by the model.

---

<div class="post-metadata">

**Author:** ![paulleopardi](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paulleopardi/32/92_2.png) [@paulleopardi](https://forum.access-hive.org.au/u/paulleopardi)\
**Post date:** [14 February 2024 04:00 UTC](https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/8 "2024-02-14T04:00:57Z")

</div>

Running the model from a restart file with the corrupted fields removed causes the model to crash with

```auto
 *********************************************************************************
 UM ERROR (Model aborting) :
 Routine generating error: INITIAL
 Error code: 1
 Error message: 
ADDR_CHK : Mis_match in start addresses
 *********************************************************************************

```

so I will only remove the corrupted fileds as a last step before comparing files. If I do this, then `mule-cumf` still gives warning messages, but does not crash.