AUS2200 vegetation fraction ancil creation issues

Hi All

@cchambers and I are attempting to run a shifted domain AUS2200 configuration, and we’re having issues around the creation of the vegetation fraction ancils. What seems to be happening is that one of the plant functional types has a huge number of unresolved points (~27k) out to a search radius of nearly 700 grid cells. As the cost of the search goes by npoints*radius**2, we’re finding that even if we set this job to the maximum walltime of 48 hours on Gadi, it won’t finish if we continue to request seasonal ancils. A quick and dirty way to get this done would be to limit it to a single month as that’s all we need for this simulation, but that isn’t a viable solution in the long term. This job takes about 10 minutes for the standard AUS2200 domain. What would the reason for such a drastic change in runtime be, and what are some potential ways to speed this up?

2 Likes

Are you generating the ancillaries with the regional ancillary suite?

Yep, the AUS2200 version thereof, u-cp145. The only change we’ve made from the version in the repo is shifting the centre longitude to 145E.

Just a guess - potentially islands that are resolved in the land mask but not in the vegetation master data. do you have coordinates of any of the misbehaving points?

2 Likes

That seems likely, the MODIS dataset is at 4km and we’re working with a 2.2km land mask derived from SRTM orography. Unfortunately the ancil code linearises the data, so all I know is that some points around 3,039,384 seem to take the longest to resolve. If my maths is right, that’s about 26.1S, 156.7E, which was outside of the original AUS2200 domain Nope, that’s inside the original domain. I’m having a look now to see what is (or isn’t) there in the ancils.

My maths was completely wrong, point 3,039,384 corresponds to 29.0S, 167.9E, and there is a little island Norfolk Island there which is completely missing from the MODIS dataset.

3 Likes

A quick update before the long weekend, I’ve made a test land mask without Norfolk Island and it did not solve the issue. In the interest of expedience @cchambers has shifted the domain centre to 140E which appears to remove the area causing the problem, so that gives us an idea where the issues lie. There is something more to it than just missing vegetation data, as this only happens on the second plant functional type. In fact, I can see exactly when it manages to resolve Norfolk Island at a search radius of 321 grid points, corresponding to about the distance between it and New Caledonia (~750km). At that point, however there are still 27,463 points remaining unresolved.

I looked at this a bit further on Thursday evening, and it turns out that the issue is New Zealand. This is the shifted domain @cchambers has been attempting to use.
Aus2200_shifted
I’ve verified that the 27,463 unresolved points correspond to the bit of New Zealand visible in the bottom right of the image. As far as I know, this only happens with the second plant functional type. The first completes quite quickly as only the Norfolk island points are unresolved after a few steps. I’ve been trying to figure out exactly where the plant functional types data comes from, I think its this dataset here: /g/data/access/TIDS/UM/ancil/atmos/master/vegetation/cover/igbp/v2/qrdata.igbp, but its not in a format I’m familiar with. Any clues on how to work with this data would be very much appreciated.

From decoding the cap source:

from typing import BinaryIO
import numpy

def read_igbp(f: BinaryIO):
    """
    Reads UKMO IGBP master data file
    """
    igbp = {}
    igbp['points_lambda_srce'] = int(f.read(5))
    f.read(1)
    igbp['points_phi_srce'] = int(f.read(5))
    igbp['phi_origin_srce'] = float(f.read(6))
    f.read(1)
    igbp['lambda_origin_srce'] = float(f.read(6))
    f.read(1)
    igbp['delta_phi_srce'] = float(f.read(10))
    f.read(1)
    igbp['delta_lambda_srce'] = float(f.read(10))
    igbp['data'] = numpy.frombuffer(f.read(igbp['points_lambda_srce'] * igbp['points_phi_srce']), dtype='b').reshape((igbp['points_phi_srce'], igbp['points_lambda_srce']))

    return igbp

Class codes:

      SELECT CASE (CIGBP_CLASS)

      CASE('A')                      ! EN forest
        IGBP_CLASS=1
      CASE('B')                      ! EB forest
        IGBP_CLASS=2
      CASE('C')                      ! DN forest
        IGBP_CLASS=3
      CASE('D')                      ! DB forest
        IGBP_CLASS=4
      CASE('E')                      ! mixed forest
        IGBP_CLASS=5
      CASE('F')                      ! closed shrub
        IGBP_CLASS=6
      CASE('G')                      ! open shrub
        IGBP_CLASS=7
      CASE('H')                      ! woody savannah
        IGBP_CLASS=8
      CASE('I')                      ! savannah
        IGBP_CLASS=9
      CASE('J')                      ! grassland
        IGBP_CLASS=10
      CASE('K')                      ! wetland
        IGBP_CLASS=11
      CASE('L')                      ! cropland
        IGBP_CLASS=12
      CASE('M')                      ! urban
        IGBP_CLASS=13
      CASE('N')                      ! mosaic
        IGBP_CLASS=14
      CASE('O')                      ! snow and ice
        IGBP_CLASS=15
      CASE('P')                      ! barren
        IGBP_CLASS=16
      CASE('Q')                      ! water
        IGBP_CLASS=17
      CASE('R')                      ! open sea
        IGBP_CLASS=18
      CASE('Z')                      ! missing data
        IGBP_CLASS=19
      CASE DEFAULT
        IGBP_CLASS=19
1 Like

So after further digging, what @MartinDix said in our meeting a couple of weeks ago was correct, ANTS is used to create the veg_frac ancil, however, CAP is still used to create the soil ancil. CAP’s workflow is veg_frac/veg_func → land ice correction → soil, so this whole thing is getting hung up on an ancil that isn’t going to be used. The only dependency I can see between the vegetation and soil calculations is the land ice mask which has to be computed after all of the vegetation functional types are resolved and after any additional glaciers are added in. However, looking at the namelist values for this calculation, No land ice processing is ever done, so I think its safe to leave points in the vegetation ancils unresolved and skip straight to the soil ancil. I’ve tested this out on the original AUS2200 domain that doesn’t hang and it has created and identical qrparm.soil ancil. It still feels like a workaround rather than a solution, as CAP needs to be modified and it won’t work as soon as glaciers become a factor. As far as I can tell, soil parameters are still generated by CAP in RAL3.2, so perhaps workarounds are all we have until ANTS fully replaces CAP, which is either a work in progress or fully completed depending on which ticket you look at in the RMED space on MOSRS.

2 Likes

Hi all.

Just bumping this because we’ve found similar issues when attempting to use Chermelle’s regional nesting suite over Gippsland and the Tasman Sea.

Scott - can you point me to the source code where you grabbed the IGBP classifications?

Thanks

https://code.metoffice.gov.uk/trac/ancil/browser/main/trunk/src/code/igbp_veg/select_igbp_class.F

1 Like

@Paul.Gregory have you tried the ants.analysis.make_consistent_with_lsm function?
https://code.metoffice.gov.uk/doc/ancil/ants/0.16/analysis.html#ants.analysis.make_consistent_with_lsm

I haven’t tried this with soil ancils, but seems to do the job with land_frac.

No I have not!

Thanks for pointing that out to me, I will try it out.

Method would be:

import ants
source_cube = ants.load_cube(source_path)
target_lsm = ants.load_cube(target_lsm_path)
ants.analysis.make_consistent_with_lsm(source_cube, target_lsm, invert_mask=True)
1 Like

Ok I’ve determined the issue for Andrew Brown’s regional nesting suite over the Tasman Sea is caused by missing values of canopy height for Macquarie and Campbell Islands.

There are no values (NaNs) for pseudo levels 1 and 2, but there are values for levels 3,4, and 5.

This suggests that the problem could be fixed within the canopy height calculation itself, rather than fixing these issues post-reconfiguration.

We believe it’s a similar issue that afflicted Chris Chambers’ shifted AUS2200 suite (which started this thread) - I’m just waiting for some extra scratch space on gadi to regenerate those bad ancillaries.

Canopy heights are computed within a rose/cycl task in a reconfiguration suite, e.g. task app/ancil_canopy_heights in suite u-dg767

The python code used to compute canopy height is installed at ~/cylc-run/u-dg767/src/contrib/Apps/CanopyHeights/ancil_canopy_heights.py

From the source code comments in ancil_canopy_heights.py

"""
Derive the canopy heights from the leaf area index by the following relation::
 
    Canopy height = height factor * LAI^(2/3)
 
Following this, override all tree PFTS using a trees dataset by the following
method:
- Regrid the tree source dataset to the target grid.
- Make this field consistent with the landsea mask of the LAI by performing an
  indexed based nearest neighbour search, constraining y to a distance of 50km
  (100km latitude band).
- Override tree fields we calculated from the above relation with those derived
  from this tree source.
The canopy heights returned represents the maximum value across the year
assigned to each month, due to the current basic handling of seasonal
variability.
"""

The python executable is called with the following command:

python_env \${CONTRIB_PATH}/Apps/CanopyHeights/ancil_canopy_heights.py ${lai} \
       --canopy-height-factors ${canopy_height_factors} --trees-dataset ${trees} \
       -o ${TMPDIR}/canopy_heights.nc --ants-config ${ANCIL_CONFIG} &&
       python_env ancil_2anc.py \
       ${lai} ${TMPDIR}/canopy_heights.nc -o ${output}

where

lai=/home/548/pag548/cylc-run/u-dg767/share/data/ancils/Gippsland/era5/lai.nc
canopy_height_factors=/g/data/access/TIDS/UM/ancil/data/transforms/canopy_height_factors_gl9.json
trees=/home/548/pag548/cylc-run/u-dg767/share/data/etc/ancil_master_ants/vegetation/
      canopy/simard/v1/Simard_Pinto_3DGlobalVeg_JGR.nc

The resulting output netCDF file is than added to the ancilary file qrparm.veg.func.

See here for more information :

Examining the canopy height fields in the Tasman Sea
How I found the issue:

Checking UM ancillaries notebook

Before I debug this at the python source level, has anyone examined this issue before?

These results may be sensible, in that Macquarie Island is essentially tree-less. But we’d have to replace NaN with some value (zero?).

@AndyHoggANU and @mlipson - does this fall into the kinds of work you are doing re: creating CABLE ancillaries for Regional nesting suites?

I’ll also circulate this to people at BoM.

Just thought I’d check that I’m not duplicating exisiting work before I dive in.

Hi Paul,

This does fall into the kind of work I’m currently doing, but I haven’t generated ancils below ~45 degrees, so haven’t encountered this issue.

I am currently examining ancillary generation with a slightly newer version of ANTS. If you can provide your grid.nl text file from your ancil directory, I can see if the same problem persists in my workflow.

Hi Paul,

Looking into this in a little more detail, I can see while there are some changes in my version of ANTS, they would not address the problem you are seeing.

My guess is that the source file used by trees variable:

/g/data/access/TIDS/UM/ancil/atmos/master/vegetation/canopy/simard/v1/Simard_Pinto_3DGlobalVeg_JGR.nc

does not include small islands like Macquarie, so in the ANTS equation Canopy height = height factor * LAI^(2/3) you are getting NaN.

edit: Actually it’s probably this line in ancil_canopy_heights.py where they appear to average their calculation for canopy height based on LAI with this Simard dataset?

trees = decomp.decompose(ants.analysis.mean, trees_cube, canopy_heights_cube[0])

The LAI source dataset has valid values in the vicinity of Macquarie Island i.e. '/g/data/access/TIDS/UM/ancil/atmos/master/vegetation/lai/modis_4km/v2/lai_preprocessed.nc', so I don’t think that file is the problem (although I don’t have your domain details).

This does look like a python/ ANTS fix that should be fed back to the MO trunk, as it will keep coming up as we move to higher resolution regional/global configurations.

Hi All,

I have been working on a way to deal with the missing canopy height pixels in the u-dg768 suite, to enable the suite to run.

Just for recap, the u-dg768 suite is a double level nest strategy that ACCESS-NRI has been working on to enable higher-resolution land/surface information to be used in regional nesting suite runs based off ERA5. The ACCESS-NRI suggested strategy is to run at the resolution of the higher-resolution land/surface information (i.e. every 0.1 degree for era5-land), replace the land/surface initial conditions, then run another nest at the target resolution.

I looked into getting the @andrewb1 double-nested domains to work.

In terms of the missing canopy heights found by @Paul.Gregory, in the southern ocean for a domain that has grid-points matching era5-land (i.e. every 0.1 degree) there are two grid-points affected. In this case only the broad-leaf plant type has a missing (nan) canopy height. When a canopy height is supplied the forecast successfully runs.

To check for other problematic issues in the matching era5-land (i.e. every 0.1 degree) Australian domain I ran a domain that extends from 11.9N to -64S and found that there are other problematic grid-points that relate to mismatches in the coastlines. These are beyond the canopy height issue and relate to land/sea issues near coastlines raised by @MartinDix.

Filled blue circles highlight all problematic pixels.

I have worked on a way to diagnose and fix these types of grid-point problems in the u-dg768 pipeline. In my prototype system the 11.9N to -64S forecast successfully runs.

We are working to include these changes into the u-dg768 (non-prototype) pipeline.

This does not address the spiral search issue highlighted by @dale.roberts and @Paul.Gregory for the AUS2200 set-up. It may though provide an interim way forward for larger double-nested domains if the high-resolution domain does not include areas affected by spiral search issues currently being discussed.

Best regards,

Chermelle

1 Like

Thanks Chermelle, looking forward to a general solution. For the subset of the missing canopy height problem I have a suggested one-line update.

In the file:
~/cylc-run/u-dg767/src/contrib/Apps/CanopyHeights/canopy_heights.py
increase the neighbourhood search radius from 500 km to, say 2500km:

loop_lim_y = index_nearest_neighbour.ydist2index(trees, 2500)

This didn’t seem to slow down the ancil generation process (full app runtime was still only 30 seconds).

Results before and after, with problem cells circled in red.
Before:


After:

This also works for high-resolution domains.
Before:


After:

3 Likes