Early ACCESS-ESM1.6 ILAMB Evaluation Pipeline Using ACCESS-MOPPy

CMIP7 CMIP7 evaluation
, , ,
1

We would like to share an early ACCESS-MOPPy + ILAMB workflow for CMORising and evaluating ACCESS-ESM1.6 historical runs using ILAMB.

Documentation for the workflow is available here:
https://access-moppy.readthedocs.io/en/latest/CMORise_ILAMB_workflow.html

The main objective of this workflow is twofold:

  • Evaluate ACCESS-ESM1.6 historical production runs using ILAMB.
  • Validate that ACCESS-MOPPy is correctly handling CMORisation and derived variable generation.

As part of this effort, @RhaegarZeng has been running tests on historical ACCESS-ESM1.6 runs and comparing the results against ACCESS-ESM1-5 for the variables currently supported by ILAMB. The resulting ILAMB outputs are now publicly available at:

/g/data/zv30/public/data/ILAMB_ESM16_Evaluation

We would greatly appreciate community feedback on these outputs. In particular, we are interested in identifying:

  • suspicious model behaviour,
  • CMORisation issues,
  • incorrectly derived variables,
  • metadata inconsistencies,
  • or anything else that looks unusual.

Please keep in mind that the workflow is still somewhat rough around the edges. You may notice a mixture of CMIP6 and ACCESS-ESM1-5 references throughout the workflow and outputs. This is mostly due to the fact that ACCESS-ESM1.6 has not yet been formally registered within the CMIP7 controlled vocabularies.

At the moment, the workflow focuses on variables supported by ILAMB, but we can extend the coverage if additional variables are needed.

ACCESS-MOPPy is under active development and regularly updated. The latest release is available through the latest version of the analysis3 conda environment on NCI (currently analysis3-26.05).

If you encounter issues or spot anything unexpected, please feel free to open an issue on the ACCESS-MOPPy GitHub repository:

https://github.com/ACCESS-NRI/ACCESS-MOPPy

If there are other ACCESS-ESM1.6 runs that you would like evaluated using this workflow, please contact either @RhaegarZeng or myself @rbeucher directly.

Over the coming months, we will also be developing training materials and video tutorials to help the community use ACCESS-MOPPy directly, particularly once ACCESS-ESM1.6 registration is complete and proper CMIP7 CMORised outputs become available.

Thanks, everyone, and we look forward to your feedback.

2

Hi @rbeucher and @RhaegarZeng ,

I’ve finally had a chance to look through the example results for ILAMB you shared.

A few questions I have:

  1. Is it expected that this example only shows ACCESS-ESM1.6 data from ~1950 onwards? While the ACCESS-ESM1.5 data is from 1850 onwards. E.g. see the time-series in EcosystemandCarbonCycle/GlobalNetEcosystemCarbonBalance/Hoffman/
  2. How is the surface albedo being derived from the ACCESS-ESM1.6 output?
  3. I wonder if there is a land mask issue when processing the ACCESS-ESM1.6 data. In a number of cases there a real values over the ocean which may be contaminating some of the ILAMB benchmarking metrics. For example:
    1. The Leaf Area Index for ACCESS-ESM1.6 seems to have real values over the ocean, whereas the benchmark data and ACCESS-ESM1.5 do not. See EcosystemandCarbonCycle/LeafAreaIndex/MODIS/. This issue is probably causing the apparent low model LAI in the relationship plot against Precipitation (see figure “ACCESS-ESM1_6_historical_Moppy_cmorised_global_rel_func_Precipitation.png”).
    2. Same issue for the NitrogenFixation/Davies-Barnard/.
    3. Same issue for Biomass/ESACCI/.
  4. Why aren’t ACCESS-ESM1.5 results reported for NitrogenFixation?
  5. I can see there is FLUXCOM benchmark data being used EcosystemRespiration. There should also be FLUXCOM data available for for GrossPrimaryProductivity. Possibly also NetEcosystemExchange. Can you include those please? And I believe there may be multiple variants
  6. Can you also please include the WECANN data for GrossPrimaryProductivity?
  7. Can you also please include the XuSaatchi2021 data for Biomass?

Finally, on the next test run of the workflow it would be more insightful for us if a more recent historical run was used. The run shared here is flagged for use as the first production run of the emissions-driven historical: /g/data/p66/rml599/historical/esm-historical-01/

Many thanks!

3

Thank @alexnorton for your time on this.
That’s really useful. @RhaegarZeng and I will have a look and get back to you next week.

5

Thanks @alexnorton, we appreciate your useful feedback. I’ll take a look and rerun the experiment with additional historical runs.

6

I am updating the observation dataset collection to address 5, 6, and 7.
Number 1 is easy and we will provide an update ASAP

2, 3, and 4 nee a bit more time. Not sure what is happening with the land mask

7

Surface albedo is not a direct ACCESS-ESM1.6 diagnostic. MOPPY derives it in two steps:

  1. rsds (surface downwelling SW) is taken directly from fld_s01i235.
  2. rsus (surface upwelling SW) is computed as fld_s01i235 − fld_s01i201 (downwelling minus net SW).
  3. Albedo is then rsus / rsds, following ILAMB’s approach, with values masked where rsds < 10 W/m².
8

Thanks Romain. I’ll refer @inh599 to this calculation of albedo, as he can assess this better than I can.

9

This MOPPY approach is appropriate to estimate surface albedo from ESM1.6.

There is a bit of a question around whether it would be better/different to use go via tiled net radiation diagnostics (i.e. evaluate rsus_tile as rsds + rlds - rlus_tile - netrad_tile and then average over tiles, since that will capture the diurnal cycle better), and/or to do the time average of rsus/rsds (not hte ratio of the averages).

In either case I suspect that we’d have to be very careful about exactly when the output is being a) evaluated in the time stepping and b) averaged over internally - so better to stick with what is there and treat with caution.

10

Hi @alexnorton,

Thank you very much for your suggestions and detailed feedback.

The recent ILAMB version update introduced a number of unexpected issues into the workflow, so it took us a little longer than expected to complete another round of testing. The new evaluation results are now available at:

/g/data/zv30/public/data/ILAMB_ESM16_Evaluation/ilamb_result/_build_07_07

Please have a look when you have a chance.

Regarding the points raised in your previous feedback, we have addressed most of them in this new run. The main remaining issue is the potential land mask problem, which we are still investigating and discussing internally to determine the best long-term solution.

Thank you again for all the valuable feedback. It has been extremely helpful in improving both the workflow and the evaluation outputs. We look forward to hearing any further comments or suggestions after you’ve had a chance to review the updated results.

11

Hi @RhaegarZeng and @rbeucher , thanks so much for this update. Can you confirm for me what ACCESS-ESM1.6 run was used to generate these results?

It would be useful to run at least two ACCESS-ESM1.6 runs through ILAMB:

  1. A concentration-driven historical run (/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/production/ensemble-concentrations-historical/historical-historical-1.1-r1i1p1f1-01409bc3/)
  2. An emissions-driven historical run (/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/production/ensemble-esm-historical/esm-historical-esm-historical-1.1-r1i1p1f1-135fafe6/)

At some point it would also be useful to include an AMIP run, but they’re not produced yet.

12

Hi, @alexnorton

I have rerun the ILAMB tests with those two ACCESS-ESM1.6 runs. The outputs are stored in: `/g/data/zv30/public/data/ILAMB_ESM16_Evaluation/ilamb_result/_build_07_09`

We used following ACCESS-ESM1.6 runs in the ilamb test run:

/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/production/esm-historical-01

/g/data/p66/rml599/historical/esm-historical-01/

/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/production/ensemble-concentrations-historical/historical-historical-1.1-r1i1p1f1-01409bc3/

/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/production/ensemble-esm-historical/esm-historical-esm-historical-1.1-r1i1p1f1-135fafe6/
13

Perfect thanks so much Rhaegar.

@clairecarouge could we put these most up-to-date results up on ME.org please? Thanks

14

@RhaegarZeng and @rbeucher , a couple of issues we’ve noticed.

  1. The GlobalNetEcosystemCarbonBalance model data is likely not being calculated correctly. We suspect land-use change emissions are not being included when it should be. Can you clarify how you calculated this from the model output? There are a couple of ways to do it. Must also be careful with units as some are flux of C while others are flux of CO2.
  2. The data for NetEcosystemExchange for all ESM1.6 results show near zero mean flux. Something has gone awry here, perhaps an incorrect time-period? Or results were taken from a piControl run by mistake?
  3. Given point 2, it is worth checking all other ESM1.6 results as well to ensure the correct run and time-period are being selected. If NetEcosystemExchange is wrong, I suspect possibly a few others could be too, especially EcosystemRespiration and GrossPrimaryProductivity.

Including @RachelLaw and @clairecarouge here as they commented as well (see here).

Cheers

15

@alexnorton Considering the questions on the current results on me.org, should we wait before we put some more results up? Or at the opposite, would it help to have the latest on as well?

16

I think we wait, thanks Claire

17

Hi All,

Thanks for the comment.
I think this is related Land fraction weighting in GPP, NPP and respiration · Issue #115 · ACCESS-NRI/ACCESS-MOPPy · GitHub
My understanding is that ESM1.5 had some issues.

Here is what I did a few weeks ago: Land fraction weighting in GPP, NPP and respiration · Issue #115 · ACCESS-NRI/ACCESS-MOPPy · GitHub

And the current calculatioin

\mathrm{nbp} = \frac{ \mathrm{fld\_s03i262} - \mathrm{fld\_s03i293} - 3.1688087814029657\times10^{-14} \, \mathrm{weighted\_tile\_sum} \left( \mathrm{fld\_s03i907} + \mathrm{fld\_s03i908} + \mathrm{fld\_s03i909}, \; \mathrm{tilefrac}=\mathrm{fld\_s03i317}, \; \mathrm{landfrac}=\mathrm{fld\_s03i395} \right) }{ \mathrm{fld\_s03i395} }
18

I used Ian’s and Martin comments

19

The emission driven case is not handled yet by Moppy
Feel free to help here:

20

@rbeucher - as noted in the github issue - I probably mis-led the thinking on whether emissions-driven and concentration-driven runs need to be done differently. As long as all the fields are available in the output (which they will be with the current STASH that we are using), both methods should give the same answer. The one using fld_s03i100 is much simpler. It would be good to implement that to double-check whether the more complicated version using the wood respiration is correct.