ACCESS-OM3 evaluation

COSIMA
,
1

The purpose of this topic is to discuss and plan the evaluation of ACCESS-OM3.

Below I’ve summarised the evaluation metrics and diagnostics that were included in the ACCESS-OM2 description paper (Kiss et al. 2020) and technical report (Kiss et al.). I’ve done this as a poll so that you can vote on your favourites. Links to code are included where I could find them.

This topic has post voting enabled. Please reply with evaluation metics/diagnostics that you would like to see included in ACCESS-OM3 evaluation. Please also up/down-vote metrics/diagnostics suggested by others.

ACCESS-OM2 evaluation

The following were used to evaluate ACCESS-OM2 as part of the ACCESS-OM2 description paper (Kiss et al. 2020) and the ACCESS-OM2 technical report.

In Kiss et al. 2020
  • Time series of global average annual mean:
    • ocean temperature
    • sea surface temperature - compared to ERSST v4
    • sea surface salinity
    • ocean kinetic energy
    See Fig 3 and and code here.
  • Time series of annual mean transport though Drake Passage. Compared to observational estimate from Donohue et al. (2016). See Fig 4 and code here.
  • Contours (latitude-longitude, global) of 1993-2012 mean dynamic sea level. Compared to CNES-CLS13. See Fig 5.
  • Contours (latitude-longitude, global) of 1993-2012 standard deviation of sea level anomaly. Compared to AVISO SSALTO/DUACS. See Fig 6.
  • Contours (potential density-latitude) of 1993-2017 mean zonally integrated overturning circulation. See Fig 7.
  • Time series of AMOC. Compared with observational estimate of McCarthy et al. (2015). See Fig 8.
  • Contours (depth-time) of horizontally averaged temperature anomaly relative to WOA13. See Fig 9.
  • Contours (latitude-longitude, global) of 1993-2017 mean sea surface temperature bias relative to WOA13. See Fig 10.
  • Contours (latitude-longitude, global) of 1993-2017 mean sea surface salinity bias relative to WOA13. See Fig 11.
  • Contours (depth-latitude) of zonally averaged temperature and salinity bias relative to WOA13. See Fig 12 and code here.
  • Lines of total meridional heat transport as a function of latitude. Compared to NCEP and ECMWF reanalyses and World Ocean Circulation Experiment. See Fig 13.
  • Meridional transects of 1993-2017 mean potential temperature and salinity near WOCE/GO-SHIP hydrographic lines:
    • SR3 (Fig 14)
    • P16 (Fig 20)
    • A16 (Fig 23)
    • IO8 (Fig 25)
    • IO9 (Fig 25)
    Compared to climatologies (1985-2013) from WOA13. See code here.
  • Meridional transects of 1993-2017 mean planetary geostrophic potential vorticity, potential density anomaly and max/min monthly mean mixed layer depth across a Subantarctic Mode Water formation region at 120W. See Fig 15.
  • Contours (latitude-longitude, regional) of 1993-2017 standard deviation of sea level anomaly and mean barotropic streamfunction in the:
    • Agulhas region (Fig 16)
    • East Australia Current region (Fig 17)
    • Kuroshio region (Fig 21)
    • Gulf Stream region (Fig 22)
    Compared to AVISO SSALTO/DUACS (sea level anomaly) and observational estimate of Colin de Verdiere and Olltrault (2016) (barotropic streamfunction).
  • Time series of annual mean transport through the Indonesian straits. Compared to estimates from INSTANT programme. See Fig 18.
  • Contours (depth-longitude @ equator, depth-latitude @ 220E) of temperature and zonal velocity. Compared to observational estimate from Johnson et al. (2002). See Fig 19.
  • Contours (latitude-longitude, SW Atlantic) of 1993-2017 mean surface current speed, standard deviation of sea level anomaly and mean barotropic streamfunction. Compared to drifter data (surface current speed), AVISO SSALTO/DUACS (sea level anomaly) and observational estimate of Colin de Verdiere and Olltrault (2016) (barotropic streamfunction). See Fig 24.
  • Contours (latitude-longitude, global) of annual 1993-2017 mean depth of the 20C isotherm. Compared to WOA13. See Fig 26.
  • Time series of running 12-month minimum, mean and maximum sea ice extent and volume for the Arctic and Antarctic. Compared to NSICD Sea Ice Index version 3. See Fig 27.
  • Lines of 1993-2017 mean annual cycle of sea ice extent for the Arctic and Antarctic. Compared to NSICD Sea Ice Index version 3. See Fig 27.
  • Contours (latitude, longitude, regional) of 1993-2017 sea ice thickness and concentration around:
    • Arctic for March and Sept (Fig 28)
    • Antarctica for Sept and Feb (Fig 29)
    Compared to NOAA G02202 V3.
0 voters
In the technical report (and not in Kiss et al. 2020)
  • Contours (latitude-longitude, global) of mean barotropic streamfunction. See Fig 13.
  • Contours (latitude-longitude, Antarctica) of Antarctic Circumpolar Current barotropic streamfunction. See Fig 14.
  • Contours (latitude-longitude, regional) of daily and climatological mean surface current speed in the:
    • East Australia Current region (Fig 17, 21)
    • Agulhas region (Fig 18, 22)
    • Kuroshio region (Fig 19, 23)
    • Gulf Stream region (Fig 20, 24)
  • Time series of annual mean transport through the Bering and Denmark straits. See Fig 25.
  • Time series of global average annual mean:
    • sea surface height
    • salinity
  • Time series of running 12-month minimum, mean and maximum sea ice area. Compared to NSICD Sea Ice Index version 3. See Fig 33.
  • Lines of 1993-2017 mean annual cycle of sea ice area for the Arctic and Antarctic. Compared to NSICD Sea Ice Index version 3. See Fig 36.
0 voters
3 Likes
7

To diagnose how good the simulation of Antarctic dense water formation is, it would be really useful to have plots of:

  • Transport across the 1000 m isobath around Antarctica, summed along the isobath and as a function of density.
  • Surface water mass transformation integrated over the Antarctic continental shelf.
4

Couple of points regarding the sea ice concentration:

  1. we should now be using v4 of the NSIDC CDR product, not v3 (NOAA/NSIDC Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 | National Snow and Ice Data Center)

  2. Can’t see the code for figs 27-29, but strictly speaking Walt Meier (NSIDC/NASA) recommends masking out model grids with a thickness less than 20mm/2 cm (the passive microwave algorithm struggles with ice this thin so it’s not a fair comparison with obs)

1 Like
5

For wave-ice penetration – I can (soon!) provide wave penetration into the MIZ for 2013 - present with AltiKa. Going back before 2013 is possible with other instruments but I can guarantee the 2013-present period.

For sea ice motion – Tian’s work showed that OSI-405c is basically as good as anything (large-scale) for sea ice drift (48 h) – but not for short timescale (any EVP model is probably poor at short timescale). https://osi-saf.eumetsat.int/products/osi-405-c

For thickness I guess the new ESA CCI is probably the best long-term one - Sara Kacimi might have better input here! The new CCI will be available soon from the CCI Data Portal (version 3). Version 2 is available but the ESA guys don’t have much confidence in it (compared to v3).

2 Likes
9

There are also things that should be checked to catch model configuration errors in earlier stages of development, e.g. timeseries of min and max of SSH, SST, SSS to find crazy behaviour, maps of time-mean SSS restoring, conservation of water between components, etc.

17

At the recent ocean team workshop we’re curating a list of datasets that will be used by many people in the community and be available using ESMvaltool/re-gridded to om3 grids for easy comparison.

We’d like feedback on the best/most up to date datasets.

We now have (updated 21st Jan 2024 based on comments – below):

Perhaps @RyanHolmes, @Pearse, @Matthis_Auger, others might have some suggestions?

@taimoorsohail and @schmidt-christina in Hobart recently, we talked about T/S alternatives?

1 Like
3

Just to note that we (@aekiss or I) will have the old code for other figures somewhere – but we weren’t able to keep them up to date with cookbook developments, so they might need some renovation to bring back up to date. Let me know if you want me to dig them out and push them somewhere.

1 Like
6

Apologies @AndyHoggANU, @anton and @willrhobbs, I’ve moved the poll into my first post since, as pointed out by @Aidan, the poll was slipping down in visibility due to the post voting. Would you be able to do the poll again please?

2 Likes
8

For equatorial/tropical Pacific verification, I would suggest the following:

  • Vertical profiles of temperature and zonal velocity compared to observations from the TAO array at key locations (e.g. 140W and 110W on the Equator). Example plots are already in COSIMA recipes (see the “Plot vertical profiles at the Equator compared to TAO, Johnson and WOA13 data” section of this notebook). The Johnson data set is pretty old now and comparing with it suffers from the issue of non-matching temporal periods (a particular issue in this region given the large internal variability). The comparison with TAO can be done robustly by restricting to times when observations exist (I have some code to do this somewhere, can dig it up if needed).

  • Equatorial slices of temperature bias compared to WOA13 (see “Plot longitude-depth temperature bias to WOA13” of this notebook, also see this notebook and Fig. 12 of Holmes et al. 2021). This will be really interesting to compare with the original ACCESS-OM2 biases because we expect that ACCESS-OM3 (depending on the vertical coordinate used) may have reduced spurious mixing that seems to have a big impact on the equatorial thermocline structure (Holmes et al. 2021).

  • Plots of Tropical Instability Wave SST and SSH variance compared to satellite observations. TIWs have a big impact on the equatorial heat budget and are particularly weak in ACCESS-OM2, and in fact in most models in OMIP-2, as described/summarized in this notebook/write-up. Again, the different numerics in ACCESS-OM3 may result in different TIWs.

2 Likes
10

Hi all,

As a follow on from work done during the ACCESS-NRI CMIP7 hackathon held in Aspendale earlier in March 2024, I’m publishing here an editable table of the key metrics we want to use to evaluate ACCESS-OM3.

Please feel free to edit and add more metrics, but please do not delete existing metrics. Instead of deleting, I suggest downgrading (or upgrading) the priority of the metric.

1 = upmost priority (i.e., sanity checks; is the model working?)
2 = essential for performance and evaluation (i.e., embarrassing if not right)
3 = of secondary importance (i.e., we can live without these being right)

Diagnostic Realm Units Dims Reference dataset Priority
Global mean temperature Ocean ºC 1D (time) WOA, Levitus 1
Global mean salinity Ocean psu 1D (time) WOA, Levitus 1
Global min/max temperature Ocean ºC 1D (time) WOA, Levitus 1
Global min/max salinity Ocean psu 1D (time) WOA, Levitus 1
Global mean temperature at 4 depth layers (0,10,100,1000m) Ocean ºC 1D (time) WOA, Levitus 1
Surface mean temperature Ocean ºC 1D (time) WOA, Levitus, ERSSTv4 1
Surface mean salinity Ocean Psu? 1D (time) WOA, Levitus 1
Mean kinetic energy Ocean 1D (time) ? 2
Drake Passage transport Ocean Sv 1D (time) Donohue et al., 2016 1
Mean dynamic sea level Ocean 2D (lon x lat) AVISO 2
Standard deviation of sea level anomaly Ocean 2D (lon x lat) AVISO 2
Meridional overturning stream function (global ocean) on density surfaces Ocean Sv 2D (lat x depth) ? 1
AMOC strength at 26N (max transport in the North Atlantic ocean-NADWF) Ocean Sv 1D (time) RAPID array; McCarthy et al. 2015 2
Max intensity of Antarctic upwelling circulation (AABWF) Ocean Sv 1D (time) WOA 2
Mean global temperature anomaly (~WOA) Ocean ºC 2D (time x lat) WOA 2
Mean surface temperature anomaly (~WOA) Ocean ºC 2D (lon x lat) WOA 2
Mean surface temperature Ocean ºC 2D (lon x lat) OISSTv2, HADiSST, 2
Mean ocean temperature interpolated at 4 depths (0,10,100,1000m) Ocean ºC 2D (lon x lat) per depth 2
Mean surface salinity anomaly (~WOA) Ocean psu 2D (lon x lat) WOA 2
Mean zonal temperature anomaly (~WOA) Ocean ºC 2D (lat x depth) WOA 2
Mean zonal salinity anomaly (~WOA) Ocean psu 2D (lat x depth) WOA 2
Total meridional heat transport as a function of latitude Ocean W 1D (lat) Trenberth and Caron (2001) (NCEP, ECMWF), Ganachaud and Wunsch, 2003 (WOCE) 1
Meridional temperature along WOCE/GO-SHIP hydrographic lines Ocean ºC 2D (lat x depth) WOCE, GO-SHIP 3
Meridional salinity along WOCE/GO-SHIP hydrographic lines Ocean psu 2D (lat x depth) WOCE, GO-SHIP 3
Meridional mean planetary geostrophic potential vorticity at 120ºW Ocean s-1 2D (lat x depth) WOCE, GO-SHIP 3
Meridional mean potential density at 120ºW Ocean Kg m-3 2D (lat x depth) ? 3
Maximum and minimum of the monthly mean mixed layer depth at 120ºW Ocean m 1D (lat) ? 3
Standard deviation of sea level anomaly in: Agulhas, East Australian Current, Kuroshio, Gulf Stream Ocean m 2D (lat x lon) AVISO SSALTO / DUACS 3
Tropical Pacific thermocline depth (5S~5N average; 20ºC isotherm) Ocean m 2D (lon x dep) ? 3
Barotropic streamfunction in: Agulhas, East Australian Current, Kuroshio, Gulf Stream Ocean Sv 2D (lat x lon) Colin de Verdiere & Olltrault (2016) 3
Transport through Indonesian Straits Ocean Sv 1D (time) INSTANT programme (Sprintall et al., 2009); Guo et al (GRL, 2023) 2
Mean depth of 20ºC isotherm Ocean m 2D (lon x lat) 2
Zonal velocity at Equator Ocean m s-1 2D (lon x depth) TAO TRITON, Johnson et al. 2002 2
Zonal velocity at 220ºE Ocean m s-1 2D (lat x depth) TAO TRITON, Johnson et al. 2002 2
Temperature at Equator Ocean ºC 2D (lon x depth) TAO TRITON, Johnson et al. 2002 2
Temperature at 220ºE Ocean ºC 2D (lat x depth) TAO TRITON, Johnson et al. 2002 2
Mean surface current speed Ocean m s-1 2D (lon x lat) Laurindo et al. (2017) 3
Standard deviation of sea level anomaly in the SW Atlantic (60ºW-30ºW, 60ºS-30ºS) Ocean m s-1 2D (lon x lat) AVISO altimetry 3
Mean barotropic streamfunction in the SW Atlantic (60ºW-30ºW, 60ºS-30ºS) Ocean Sv 2D (lon x lat) Collin de Verdiere & Olltrault (2016) 3
CFC11, CFC12, SF6 comparison point-to-point with hydrographic data Ocean mmol m-3 Point to point scatter GLODAPv2 2
Age tracer Ocean years OCIM (De Vries et al. 2014) 3
Timeseries of Sea ice area Maxima/Mean/Minima (SH) Sea ice km^2 1D (time) NOAA G02202 passive microwave 1
Timeseries of Sea ice area Maxima/Mean/Minima (NH) Sea ice km^2 1D (time) NOAA G02202 passive microwave 1
Timeseries of Sea ice volume Maxima/Mean/Minima (SH) Sea ice km^3 1D (time) ? 1
Timeseries of Sea ice volume Maxima/Mean/Minima (NH) Sea ice km^3 1D (time) ? 1
Sea ice extent Feb/Sep (SH) Sea ice km^2 1D (time) NOAA G02202 passive microwave 2
Sea ice extent Feb/Sep (NH) Sea ice km^2 1D (time) NOAA G02202 passive microwave 2
Sea ice concentration (SH) Sea ice % 2D (lon x lat) NOAA G02202 passive microwave 1
Sea ice concentration (NH) Sea ice % 2D (lon x lat) NOAA G02202 passive microwave 1
Sea ice feedback (volume/growth) Sea ice N/A 0D ? 3
Annual max Sea ice thickness Sea ice m 2D (lon x lat) ? 1
Surface NO3 BGC µM 2D (lon x lat) WOA 1
Surface PO4 BGC µM 2D (lon x lat) WOA 1
Surface SiOH4 BGC µM 2D (lon x lat) WOA 2
Surface dFe BGC nM 2D (lon x lat) Huang et al., 2022 2
Depth of the nitracline BGC m 2D (lon x lat) WOA 2
O2 minimum BGC µM 2D (lon x lat) WOA 2
Primary limiting nutrient (N or Fe) BGC categorical non-gridded stations Browning & Moore 2023 1
Surface Chlorophyll BGC Mg m-3 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 1
Depth integrated Chlorophyll BGC mg m-2 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 2
Depth of maximum chlorophyll BGC m 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 1
Surface particulate organic carbon BGC µM 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 1
Depth-integrated particulate organic carbon BGC g C m-2 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 2
Depth of maximum particulate organic carbon BGC m 2D (lon x lat) Sauzede et al., 2016 JGR Oceans 1
Net Primary Production BGC mg C m-2 day-1 2D (lon x lat) MODIS CbPM 2
Fraction of microphytoplankton BGC fraction 2D (lon x lat) Kostadinov et al., 2016 2
Zooplankton grazing pressure BGC day-1 2D (lon x lat) Rohr et al., (in prep) 2
Average particle flux BGC mg C m-2 day-1 3D (lon x lat x depth) Mouw et al., 2016 3
Depth-integrated nitrogen fixation rate BGC µmol N m-2 day-1 2D (lon x lat) Shao et al., 2024 3
Phytoplankton bloom duration BGC days 2D (lon x lat) OC-CCI (Nicholson et al., 2024 Earth System Science Data) 1
Phytoplankton bloom initiation date (relative change method) BGC day 2D (lon x lat) OC-CCI (Nicholson et al., 2024 Earth System Science Data) 1
Fe:C quotas of phytoplankton BGC µmol/mol non-gridded stations Ben Twining’s data 3
5 Likes
11

I am planning on starting a new repository to capture these notebooks soon. The new repository would initially have nominal reviews/documentation only, and be formalised later. The idea is to have a low barrier to changes whilst we are iterating and working with initial prototype model runs (which might not be kept very long), and figuring out the best ways of working with the intake-catalogue. Then before publication or when the notebooks are more stabilised, we could review them on mass and merge them into cosima-recipes. We would probably start with the plots in the first post, and expand per interest / datasets / new features.

1 Like
12

Random question, but curious to know if there are ACCESS-OM3-025 run administration statistics – i.e. approximately the SU cost of running the model in ‘standard configuration’ for one model-year? If not ACCESS-OM3 then maybe ACCESS-OM2-025?

Sorry for the broadside.

13

It would be worthwhile to save the fields required for a DOCN data ocean to drive CICE6-WW3 configurations. From @ezhilsabareesh8’s poster the required fields are

  • SST
  • SSS
  • surface velocity
  • melt and freeze potential
  • mixed layer depth

@lgbennetts, @NoahDay please chime in with the duration of the 0.25° WW3-CICE6 runs you’re planning and the data frequency you think will be needed (e.g. will daily data for a few years be sufficient? As these are 2d fields we could probably go to higher frequency if the duration is not too long.)

2 Likes
15

Quick Update on CMIP7 Evaluation

The IPCC is developing a Rapid Evaluation Framework to assess submissions for the AR7 Fast Track.

Below is a table summarising the minimum requirements for participating tools, including ESMValTool and ILAMB:

Screenshot From 2024-11-20 20-53-30
Screenshot From 2024-11-20 20-53-30999×574 312 KB

Current Progress:

We’ve made significant progress on a series of notebooks for ENSO evaluation, which you can find here:
:point_right: ACCESS-ENSO Recipes Repository

  • Ocean Diagnostics: I plan to develop similar notebooks to cover ocean and sea-ice diagnostics, ensuring we meet the AR7 Fast Track minimum requirements.
  • Integration with ESMValTool: These notebooks will use the ESMValCore preprocessor, making it easier to handle multi-model or large ensemble analyses once converted into ESMValTool recipes.

A strong prerequisite for this work is CMORisation.

  • @RhaegarZeng has been developing a Light CMORiser for OM2, and I’m optimistic this could be extended to evaluate OM3 without the need to CMORise, especially for things like SST …

@anton @cbull — Where should I kick this off?
Should I use the repository Anton set up, or should I create a new one?
Ultimately, I aim to integrate this work into ESMValTool.

16

Thanks for the update @rbeucher, on this

Should I use the repository Anton set up, or should I create a new one?

As you know, we’re currently discussing / pulling together our om3 evaluation strategy. We do have a few scripts in here so that could be a place for your work? What’s the @anton one?

Related, you mentioned at the workshop that you’ve been converting a few of the COSIMA recipes, where are they?

18

Any reason to pick ERSST v4 instead of ERSST v5?
v5 is available in ct11 for the 1854-2020 period.

19

For SSH, do you need daily values (original product) or monthly means suffice?

20

Dear All,

I’ve reviewed the list of observational datasets you’ve put together for evaluating OM3. Here’s a summary of what we have so far:

  • ERSST: Version 5 is available on Gadi in the ct11 collection. We also have v3b
  • SSH:
    • SEALEVEL_GLO_PHY_L4_MY_008_047 Weekly and Monthly available in kj13
    • AVISO Altimetry Antarctic: Daily and Monthly available in kj13
  • WOA2023: Already available on Gadi.
  • EN4: Currently in kj13 but looking to move it to ct11 or the COSIMA-WG storage space.
  • IAPv2.4: Available in kj13.
  • NSIDC CDR v5 (Sea Ice Concentration): Available in kj13 for both hemispheres.
  • NSIDC Sea Ice Index: Available in kj13, bobth hemispheres
  • TOA: All data stations are available in kj13.
  • Chlorophyll (Sauzede et al.): Available in kj13

Most of the data is in raw format and will need to be CMORised to facilitate comparisons with CMIP data via ESMValTool and ILAMB. This shouldn’t be too complex, but let me know if you foresee any challenges or additional requirements. We will keep a copy of the original data of course.

21

Hi @rbeucher, thanks for following up on this so promptly. Great to hear that almost all of the data we are after is already available!

I don’t imagine we need 4 as well for ersst.

I think @pearseb might have some insight about Sauzede et al.?

22

Sauzede et al. dataset can be found on the Copernicus Marine Service web portal.

1 Like