Details of our approach to evaluate the WOMBAT model.
The following are datasets used to evaluate performance (WOMBAT-lite and WOMBAT-mid). Note that not all datasets will be applicable to WOMBAT-lite. These include surface PO4, surface SiOH4, the fraction of microphytoplankton, and depth-integrated nitrogen fixation rates.
Looks awesome @pearseb!
To optimise WOMBAT-lite, we ran 256 sensitivity experiments. These experiments explored a range of different values of 19 key input parameters to the biogeochemical model.
WOMBAT-lite was run for 10 years under the JRA55 repeat year forcing (ryf) initialised from observations of nutrients (WOA23), dissolved Fe (PISCES bgc model), oxygen (WOA23), carbon (GLODAP2) and globally uniform phytoplankton, zooplankton and detritus concentrations.
The skill of these experiments relative to 15 observation-based products is shown below:
The inter-experiment standard deviations in these fields are shown below:
The inter-experiment NORMALISED standard deviations are shown below:
Some redundancy in the observation-based products is apparent. NPP is very similar to grazing pressure and thus offers the same information. Chlorophyll and POC datasets too. We therefore focussed on 8 out of these original 15 datasets to assess model performance (see below).
Based on the above results, we developed a traffic light system of model evaluation.
The approach is quantitative, involving the univariate metrics of correlation coefficient, mean bias and normalised standard deviations for key variables. The results are then categorized into a simple, easy-to-understand “traffic light” framework:
We first define thresholds for each metric that determine the categorization into green, yellow, or red. This requires knowledge of what is reasonable model skill given the observational product we are comparing too.
Our thresholds for the 8 key observation-based products:
As a first pass, we eliminated all but the model realisations that performed optimally for the nutrient limitiation (LN) data.
This left 32 experiments, all with excellent agreement to Browning & Moore (2023) but with a range of good to poor performance in the other key observations.
The following figure shows the top and worst performing experiments of WOMBAT-lite relative to the key observation-based products we are using to assess performance.
NOTE: World Ocean Atlas O2 in the low oxygen zones tends to be biased high, and the MODIS-based CbPM model is what we use for NPP, which is very high and all models apparently underestimate this.
Wow, this is awesome. My takeaway is the NPP is bad. Iron is better than WOMBAT-old … other things OK??
I think to say that the NPP is bad is to believe that MODIS CbPM productivity model is “true”. Iron is definitely better.
Thanks @pearseb, this looks like a great way to assess the model.
I was wondering
I noticed some of the ACCESS-EMS1.5 values in Ziehn et al 2020 (PI, quadratic phy mortality, prey capture efficiency) lie outside the ranges in your parameter survey - is this because the equations or units are different?
