Dear ACCESS-NRI,
I tired to use um2nc:
module use /g/data/vk83/modules
module load payu
cd /scratch/k10/dxd565/tmp/
um2nc dj863a.pm0269dec test.nc
but the output cannot be read by “xconv” → “Segmentation fault”
Can someone please help me to convert the UM-output files to netcdf?
best regards
Dietmar
Our staff that work on um2nc are both unavailable at the moment, but I will try and help.
Is it more than one file that you would like to convert or exactly that one?
Did you know that you can open UM files with xconv and convert individual fields to netcdf using the xconv application?
Another option is to open the file in iris and save the file as netcdf?
See Saving Iris Cubes — Iris 3.13.0.dev62 documentation
Iris is available in many of the existing conda environments. If you are interested in trying that out I am happy to help. I don’t have access to the k10 group though so we will have to find a way to pass over files.
Hi @Dietmar_Dommenget,
I copied your test file to project lg87 which might be more accessible to ACCESS-NRI folks:
/scratch/lg87/dkh157/dietmar_test/dj863a.pm0269dec
I also tried running the um2nc command as you did:
um2nc dj863a.pm0269dec test.nc
And this seemed to work for me. I could open the test.nc file in ncview and in xconv ok. But there were some warnings and perhaps not every field was converted correctly.
I also ran the alternative script method which is a script called conv2nc.tcl, which is also copied into that folder.
Running the following:
conv2nc.tcl dj863a.pm0269dec
Creates a netcdf file called dj863a.pm0269dec.nc
this also seems ok (variable names are a bit different). I did notice that the file sizes ended up being a bit different. I removed the netcdf compression from each of the outputs to get a proper comparison of file sizes. The resulting files are saved as:
out_convsh.nc
out_um2nc.nc
(Both have compression removed.) The “out_convsh.nc” file from my script comes out at 40 MB, while the out_um2nc.nc file comes out as 21 MB. The original UM file is 40 MB. This suggests that a fair portion of the data was never converted properly by um2nc.
I’ve tried reconverting the um file with um2nc and also ran into the same segmentation fault when opening the result with xconv. This appears to be a limitation with xconv, which can crash when a variable’s “long_name” attribute is too long. The netCDF file test.nc contains two variables with long_name attributes of length 110, field1221 and field1225 with long names product_of_effective_radius_of_stratiform_cloud_liquid_water_particle_and_stratiform_cloud_liquid_water_area_f and product_of_effective_radius_of_convective_cloud_liquid_water_particle_and_convective_cloud_liquid_water_area_f respectively.
Using nco to shorten these attributes:
ncatted -a,long_name,field1221,m,c,<shorter-name> -a,long_name,field1225,m,c,<shorter-name> test.nc shorter_name_test.nc
let me open the file with xconv. 105 characters seems to be the limit for xconv to work.
Dear Spencer,
thanks for your reply. Unfortunately, I cannot make your suggestion work.
I will leave this for now …
best regards
Dietmar
Dear David,
thanks for your reply. Great to see it works for you. No clue why it does not for me.
I will leave this this for now … may work on it again … in the far future.
best regards
Dietmar
Great catch @dkhutch. It looks like some of the data is in fact being dropped by um2nc.
This is related to um2nc’s processing of 3D variables saved on a pressure level grid. In this case, um2nc has dropped all of these variables from the output due to the required “heavyside function” variable (stash code 30301 for variables on uv-points, and 30304 for variables on t-points) being missing from the UM output file.
During a model run, if a pressure level grid point falls below ground/sea-level in a timestep, the UM set’s the output value for all variables at that point to 0. Meanwhile, the “heavyside function” variable records the fraction of time during a mean calculation period that each point fell above ground level. E.g. if a pressure level grid point fell above ground for 60% of the time during a particular month, the monthly mean “heavyside function” at that point will be 0.6.
Without any extra processing, model output on the pressure levels will have values artificially close to 0 at points that sometimes fall below ground. um2nc divides the pressure level output by the heavyside function, recovering the mean over the period of time where a point falls above the surface.
When the relevant heavyside variable is missing, um2nc’s default behaviour is to drop the dependent pressure level variables from the output. The command line option, --nomask can be used to keep the pressure level variables without doing any masking, however points at lower levels may be affected by the extra 0s. If the pressure level data at levels closer to the surface are important to the analysis and the run isn’t too far advanced, it might be worth adding the heavyside variables to the model output.
um2nc currently only warns about this when the verbose flag -v. I think it might make sense for this warning to always occur. I’ll bring this up with the rest of the team.
Converting dj863a.pm0269dec with the --nomask option produces a 24M file, still much smaller than out_convsh.nc. Both contain the same number of variables (ignoring extra bounds/dimension variables). Looping through all the variables in the um2nc --nomask output, it looks like they nearly all match the variables in out_convsh.nc. The following shows the max absolute difference between each variable in the um2nc output and the closest matching variable in out_convsh.nc, and whether there are differences in their NaNs:
theta: closest match theta
Difference: 0.0
Same NaNs: True
hus: closest match q
Difference: 0.0
Same NaNs: True
snw: closest match snowdepth
Difference: 0.0
Same NaNs: True
ts: closest match temp
Difference: 0.0
Same NaNs: True
bldepth: closest match blht
Difference: 0.0
Same NaNs: True
sic: closest match iceconc
Difference: 0.0
Same NaNs: True
sit: closest match icedepth
Difference: 0.0
Same NaNs: True
field95: closest match snowdepth_1
Difference: 0.0
Same NaNs: True
field266: closest match cldamount
Difference: 0.0
Same NaNs: True
field407: closest match p
Difference: 0.0
Same NaNs: True
field408: closest match p_1
Difference: 0.0
Same NaNs: True
ps: closest match p_2
Difference: 0.0
Same NaNs: True
field413: closest match field1411
Difference: 0.0
Same NaNs: True
field414: closest match field1410
Difference: 0.0
Same NaNs: True
field415: closest match field1413
Difference: 0.0
Same NaNs: True
field416: closest match field1412
Difference: 0.0
Same NaNs: True
ts_sea: closest match temp_1
Difference: 0.0
Same NaNs: True
ts_ice: closest match temp_2
Difference: 0.0
Same NaNs: True
field509: closest match field322
Difference: 0.0
Same NaNs: True
rss: closest match solar
Difference: 0.0
Same NaNs: True
rss_sea: closest match solar_1
Difference: 0.0
Same NaNs: True
rsdt: closest match field200
Difference: 0.0
Same NaNs: True
rsut: closest match field201
Difference: 0.0
Same NaNs: True
rsutcs: closest match field207
Difference: 0.0
Same NaNs: True
rsdscs: closest match field208
Difference: 0.0
Same NaNs: True
rsuscs: closest match field207_1
Difference: 0.0
Same NaNs: True
field1221: closest match field164
Difference: 0.0
Same NaNs: True
field1223: closest match field166
Difference: 0.0
Same NaNs: True
field1225: closest match field163
Difference: 0.0
Same NaNs: True
field1226: closest match field163
Difference: 0.0
Same NaNs: True
rsds: closest match field203
Difference: 0.0
Same NaNs: True
field1260: closest match solar_2
Difference: 0.0
Same NaNs: True
rls: closest match longwave
Difference: 0.0
Same NaNs: True
field2203: closest match longwave_1
Difference: 0.0
Same NaNs: True
clt: closest match field30
Difference: 0.0
Same NaNs: True
rlut: closest match olr
Difference: 0.0
Same NaNs: True
rlutcs: closest match csolr
Difference: 0.0
Same NaNs: True
rlds: closest match ilr
Difference: 0.0
Same NaNs: True
rldscs: closest match csilr
Difference: 0.0
Same NaNs: True
aod_ss: closest match unspecified_1
Difference: 0.0
Same NaNs: True
field2422: closest match unspecified_2
Difference: 0.0
Same NaNs: True
hfss_ice: closest match botmelt
Difference: 0.0
Same NaNs: True
hfss: closest match sh
Difference: 0.0
Same NaNs: True
tauu: closest match taux
Difference: 0.0
Same NaNs: True
tauv: closest match tauy
Difference: 0.0
Same NaNs: True
evspsbl: closest match field184
Difference: 0.0
Same NaNs: True
field3224: closest match wme
Difference: 0.0
Same NaNs: True
uas: closest match u
Difference: 0.0
Same NaNs: True
vas: closest match v
Difference: 0.0
Same NaNs: True
wnd: closest match wind
Difference: 0.0
Same NaNs: True
field3228: closest match sh_1
Difference: 0.0
Same NaNs: True
field3231: closest match sublim
Difference: 0.0
Same NaNs: True
field3232: closest match evap
Difference: 0.0
Same NaNs: True
hfls: closest match lh
Difference: 0.0
Same NaNs: True
field3235: closest match topmelt
Difference: 0.0
Same NaNs: True
tas: closest match temp_3
Difference: 0.0
Same NaNs: True
huss: closest match q_1
Difference: 0.0
Same NaNs: True
field3245: closest match rh
Difference: 0.0
Same NaNs: True
field3256: closest match field1414
Difference: 0.0
Same NaNs: True
field3257: closest match field1415
Difference: 0.0
Same NaNs: True
field3258: closest match field141
Difference: 0.0
Same NaNs: True
gpp: closest match field1388
Difference: 0.0
Same NaNs: True
npp: closest match field1389
Difference: 0.0
Same NaNs: True
resp: closest match field1390
Difference: 0.0
Same NaNs: True
evspsblveg_tile: closest match field1517
Difference: 0.0
Same NaNs: True
rlut_afterbl: closest match olr_1
Difference: 0.0
Same NaNs: True
field3370: closest match field1813
Difference: 0.0
Same NaNs: True
field3371: closest match field1814
Difference: 0.0
Same NaNs: True
tauu_0: closest match taux_1
Difference: 0.0
Same NaNs: True
tauv_0: closest match tauy_1
Difference: 0.0
Same NaNs: True
field3510: closest match field188
Difference: 0.0
Same NaNs: True
field4203: closest match lsrain
Difference: 0.0
Same NaNs: True
field4204: closest match lssnow
Difference: 0.0
Same NaNs: True
field5205: closest match cvrain
Difference: 0.0
Same NaNs: True
field5206: closest match cvsnow
Difference: 0.0
Same NaNs: True
field5209: closest match temp_4
Difference: 0.0
Same NaNs: True
prrn: closest match rain
Difference: 0.0
Same NaNs: True
prsn: closest match snow
Difference: 0.0
Same NaNs: True
pr: closest match precip
Difference: 0.0
Same NaNs: True
field5231: closest match field1134
Difference: 0.0
Same NaNs: True
field5232: closest match botmelt
Difference: 0.0
Same NaNs: True
field5269: closest match unspecified_3
Difference: 0.0
Same NaNs: True
field5270: closest match field1598
Difference: 0.0
Same NaNs: True
field5272: closest match field1934
Difference: 0.0
Same NaNs: True
field5277: closest match precip_1
Difference: 0.0
Same NaNs: True
field5278: closest match precip_2
Difference: 0.0
Same NaNs: True
field5279: closest match precip_3
Difference: 0.0
Same NaNs: True
field5319: closest match unspecified_4
Difference: 0.0
Same NaNs: True
field5320: closest match omega
Difference: 0.0
Same NaNs: True
field5322: closest match omega_1
Difference: 0.0
Same NaNs: True
field5323: closest match omega_2
Difference: 0.0
Same NaNs: True
field6201: closest match taux_2
Difference: 0.0
Same NaNs: True
snw_afterhydrol: closest match snowdepth
Difference: 0.0
Same NaNs: False
mrso: closest match sm
Difference: 0.0
Same NaNs: True
cw: closest match field271
Difference: 0.0
Same NaNs: True
mrsol: closest match sm_1
Difference: 0.0
Same NaNs: True
tsl: closest match soiltemp
Difference: 0.0
Same NaNs: True
mrlqso: closest match field1385
Difference: 0.0
Same NaNs: True
mrfsofr: closest match field1386
Difference: 0.0
Same NaNs: True
snm: closest match field1530
Difference: 0.0
Same NaNs: True
field8233: closest match field1531
Difference: 0.0
Same NaNs: True
mrros: closest match field1532
Difference: 0.0
Same NaNs: True
mrrob: closest match field1533
Difference: 0.0
Same NaNs: True
field8245: closest match field1907
Difference: 0.0
Same NaNs: True
psl: closest match p_3
Difference: 0.0
Same NaNs: True
field26001: closest match field1902
Difference: 0.0
Same NaNs: True
field26002: closest match field1904
Difference: 0.0
Same NaNs: True
field26003: closest match field1903
Difference: 0.0
Same NaNs: True
field26004: closest match field1901
Difference: 0.0
Same NaNs: True
field26006: closest match field1906
Difference: 0.0
Same NaNs: True
ua_plev: closest match u_1
Difference: 0.0
Same NaNs: True
va_plev: closest match v_1
Difference: 0.0
Same NaNs: True
wa_plev: closest match dz_dt
Difference: 0.0
Same NaNs: True
ta_plev: closest match temp_5
Difference: 0.0
Same NaNs: True
zg_plev: closest match ht_1
Difference: 0.0
Same NaNs: True
wap_plev: closest match omega_3
Difference: 0.0
Same NaNs: True
Everything looks like matches, apart from the NaNs in snw_afterhydrol/snowdepth. I’m unsure why there’s such a large difference in file size though.