How to use cumf or mule-cumf with ACCESS ESM1.5 UM output?

In testing the ESM1.5 pre-industrial run, I noticed that there is a script that builds a version of the cumf executable that matches UM vn7.3. The script is
branches/dev/Share/VN7.3_local_changes/Install_scripts/UmBuildSmllExecs within the /g/data/access/access-svn/UM_Admin repository.

I think that if we need to build this vn7.3 cumf and related executables we may need a private https://github.com/ACCESS-NRI/UM_Admin or https://github.com/ACCESS-NRI/UM_Admin_v7 repository.

The alternative would be to get an existing build of mule-cumf working correctly for the UM vn7.3 output of ESM1.5.

Before I start anything in this direction, I need to know how current users of ESM1.5 are using cumf or mule-cumf and where they obtain the executable that they use. I do know that mule-cumf is working for others and I want to understand why it isn’t working for my username pcl851.

See Create framework for extracting, verifying and updating model reproducibility and performance · Issue #1 · ACCESS-NRI/reproducibility · GitHub

See also Porting CSIRO/UMUI ACCESS-ESM1.5 ksh run script to payu and https://code.metoffice.gov.uk/doc/um/mule_utils/2019.01.1/um_utils/cumf.html

1 Like

mule-cumf should work fine for old versions, what issue are you having?

As per Create framework for extracting, verifying and updating model reproducibility and performance · Issue #1 · ACCESS-NRI/reproducibility · GitHub
I see

[pcl851@gadi-login-05 access-esm]$ pwd
/scratch/tm70/pcl851/access-esm
[pcl851@gadi-login-05 access-esm]$ module use /g/data/hh5/public/modules/
[pcl851@gadi-login-05 access-esm]$ module load conda/analysis3-23.07
[pcl851@gadi-login-05 access-esm]$ module use /g/data/access/modules
[pcl851@gadi-login-05 access-esm]$ module load um
[pcl851@gadi-login-05 access-esm]$ mule-cumf --stashmaster 7.3 archive.orig/access-esm/restart000/atmosphere/restart_dump.astart archive.build-gadi.1/access-esm/restart000/atmosphere/restart_dump.astart 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.orig/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.build-gadi.1/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/mule-cumf", line 10, in <module>
    sys.exit(_main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 1385, in _main
    comparison = UMFileComparison(um_files[0], um_files[1])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 728, in __init__
    diff_field = difference_op([field_1, field_2])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 952, in __call__
    new_field = self.new_field(source, *args, **kwargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 293, in new_field
    data1 = fields[0].get_data()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 730, in get_data
    data = self._data_provider._data_array()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ff.py", line 193, in _data_array
    data = np.fromstring(data_bytes, dtype, count=count)
ValueError: string is smaller than requested size

Are the warnings and the ValueError to be expected, or do I have a misconfiguration?

They’re not expected, but they indicate odd properties of the files rather than something wrong with mule-cumf.

Three fields have an axis in the wrong order, and four fields claim to be timeseries (lbcode 31320), which is unexpected in a restart file. Finally it wasn’t able to read the data from at least one field.

Is one of these files somewhere where I can take a look at it?

I tried again with two different files that you should have access to:

[pcl851@gadi-login-06 access-esm]$ mule-cumf --stashmaster 7.3 /g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart /g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/atmosphere/restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: /g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4935,4937,6676,6715)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: /g/data/vk83/experiments/inputs/access-esm1p5/pre-industrial/restart/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4935,4937,6676,6715)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/mule-cumf", line 10, in <module>
    sys.exit(_main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 1385, in _main
    comparison = UMFileComparison(um_files[0], um_files[1])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 728, in __init__
    diff_field = difference_op([field_1, field_2])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 952, in __call__
    new_field = self.new_field(source, *args, **kwargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 293, in new_field
    data1 = fields[0].get_data()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 730, in get_data
    data = self._data_provider._data_array()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ff.py", line 193, in _data_array
    data = np.fromstring(data_bytes, dtype, count=count)
ValueError: string is smaller than requested size

There’s a few corrupt fields. You may have more luck running the comparison on the model output files, presumably the corrupt data is ignored by the model.

You can remove the corrupt fields and get cumf to work with

mf = mule.load_umfile('/g/data/access/payu/access-esm/input/pre-industrial/start_dump/pre-industrial.astart')

out = mf.copy()

for f in mf.fields:
    if f.lbuser4 not in [155,156,3100,3101,33001,33002]:
        out.fields.append(f)

out.validate = lambda *args, **kwargs: True
out.to_file("/scratch/hc46/saw562/access-esm.astart")
mule-cumf test
$ mule-cumf /scratch/hc46/saw562/access-esm.astart /scratch/hc46/saw562/access-esm.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning:
File: /scratch/hc46/saw562/access-esm.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
  warnings.warn(msg)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

File 1: /scratch/hc46/saw562/access-esm.astart
File 2: /scratch/hc46/saw562/access-esm.astart
Files compare
  * 0 differences in fixed_length_header (with 7 ignored indices)
  * 0 field differences, of which 0 are in data

Compared 6554/6554 fields, with 6554 matches
1 Like

Thanks, Scott. I will try to run the model again using a restart file with the corrupted fields removed, to make sure that these fields are not needed by the model.

Running the model from a restart file with the corrupted fields removed causes the model to crash with

 *********************************************************************************
 UM ERROR (Model aborting) :
 Routine generating error: INITIAL
 Error code:                      1
 Error message: 
ADDR_CHK : Mis_match in start addresses
 *********************************************************************************

so I will only remove the corrupted fileds as a last step before comparing files. If I do this, then mule-cumf still gives warning messages, but does not crash.