Scope of ACCESS-NRI ACCESS-OM2 release

COSIMA
,
1

Hey COSIMA folks,

We (ACCESS-NRI) are trying to nail down some concrete timelines for an ACCESS-OM2 release in the first half of next year.

The ACCESS-NRI “value add” for COSIMA is that we would take over support of the model, freeing scarce people resources, principally @aekiss and @micael, to work on ACCESS-OM3 development, as well as disk resources for hosting important datasets.

There are some critical things we need to establish for planning purposes:

  • Do we include all three current resolutions (1°, 0.25° and 0.1°) in supported configurations?
  • Are there any anticipated updates or code changes that are necessary before an ACCESS-NRI release? (Here updates might be, but not limited to, updated inputs or configs)
  • Are there any anticipated updates or code changes that are desirable, but can come after release?
  • What control runs are considered necessary? RYF/IAF? What period/length?
  • Would ACCESS-NRI calculate it’s own control run or take over responsibility for existing COSIMA runs?
  • If an IAF control run is required, would the community expect that to be updated when new forcing data becomes available?
  • Do we need to maintain bit-reproducibility between current ACCESS-OM2 configurations and new releases?

Cheers

Aidan

2

At present we have 12 supported configurations in these 6 repositories (also included in access-om2/control):

These include

  • 3 nominal horizontal resolutions: 1° (1deg), 0.25° (025deg) and 0.1° (01deg)
  • 2 forcing types: JRA55-do data that repeats 1 May 1990 - 30 April 1991 (ryf, repeat-year forcing) or 1 Jan 1958 - 31 Dec 2018 (iaf, interannual forcing)
  • a master branch (for physics-only simulations) and a master+bgc branch (that also includes ocean and sea ice biogeochemistry) in each repository

There may also be ERA5 variants when we get that working.

3

There’s a new 0.25° topography available that fixes land mask bugs around Antarctica: Fix land mask around Antarctic coast for 0.25deg topography · Issue #265 · COSIMA/access-om2 · GitHub

We should use this in supported configs once it’s been tested.

4

This 0.25° topo fix should be included before release.

5

ERA5 support will involve code changes. Not sure if that should be done before or after release.

6

Once you’ve change the topography bit reproducibility with previous runs is off the table. I’m guessing this would also necessitate new control runs?

It would be useful to know how bit reproducibility has been handled in the past when there have been code and/or config changes. Have both the lower resolution control runs been re-done in those cases? I know the 0.1° configurations are much more expensive than the others, so have been a special case with respect to re-running.

If there is no associated control run with ERA5 forcing, and I haven’t seen this suggested as yet, then it could come after release.

Are there any other parameter values that have been revised since the configurations were last updated?

7

Perhaps we should add this to a COSIMA meeting agenda @rmholmes? In 2 weeks perhaps if tomorrow is full already.

8

@adele157 sounds good. The 8th of December slot has actually just opened up with no speaker. I feel like we could fill that one up with this discussion (and perhaps more detail from Aidan on how the Hive will work, and how NRI and COSIMA might interact on a more technical level).

9

@Aidan just wanted to confirm that we can fill up the COSIMA meeting on Thursday with this discussion? All I have on the agenda for Thursday’s COSIMA is a few minor things and:

  • ACCESS-OM2 release plan with the NRI.
  • Discussion of how COSIMA and the NRI will work together on a technical level.
10

I think we can make a decent fist of filling up the time. I do think we need to get started with some of these discussions and I’d be reluctant to wait until next year.

11

Take control of existing restarts and enable perturbations runs? How do we bridge existing with new?

Support RYF 0.1 degree.

Good reasons for updated control run: new bathymetry, basal melt? New JRA55 maybe not.

Adele: did single cycle of IAF on JRA1.5. Changed diffusivity also.

RYF: better for perturbation runs. So used for process modelling studies
IAF: community tool, used more by those accessing existing data

Would be valuable: Curation of existing restarts.

Possible new feature: tool for verifying restarts when starting new perturbation run.

IAF restarts are every 5 years, 98 and 2003, makes planning runs more difficult.

Length of control runs: 3rd cycle IAF onwards. Stabilise by 3rd cycle. Drift depends on where you’re interested in the water column. Cannot spin up abyssal ocean at all in these runs.

IAF spin-up comes from IMIP protocol. Get shock at the beginning of every run. Ocean heat-uptake data useless for first 20 years. Can use other spin-up methods, e.g. repeat-decadal forcing.

Current shallow bathymetry around antarctic coast causes model blow ups.

Rough compute costs for models:

1: ???
0.25: ???
0.1: 150-250KSU / year for 0.1 degree with BGC. 9-16MSU/cycle (65yrs)

Takeaway:

  • Take-over current configurations/control runs: facilitate gathering metadata and improve training materials for users.
  • COSIMA target some possible control run updates: good for community, also good for NRI as test bed for improved and automated provenance chains. e.g. basal melt 0.1, improved bathymetry 0.25
12

I think this would be great, along with a system for the user to know exactly what configuration, executables and input data are required to branch from that restart. We already have these systems in the git history (manifests, hashes etc.), it’s just a matter of making this transparent for a new user (currently, they would email the person who ran the control run who would then send them links to the appropriate configuration commits and restart files).

See Drivers and distribution of global ocean heat uptake over the last half century | Nature Communications for more detail. In particular Fig. 1b shows how long it can take global OHC to stabilize (solid red line, from repeat decade forcing) and the shocks you get from the current OMIP-style repeat IAF cycles approach (dark blue line):

image
image877×494 77.8 KB

Whether this is important depends on what you’re interested in.

From last year’s NCMAS application:
1: 0.15 kSU/year, 10 kSU/cycle
1+BGC: 0.24 kSU/year, 15.6 kSU/cycle
0.25: 7.2 kSU/year, 500kSU/cycle
0.25+BGC: 13.1 kSU/year, 850 kSU/cycle
0.1: 156 kSU/year, 10MSU/cycle
0.1+BGC: 250 kSU/year, 16 MSU/cycle

13

Would be good to start this process. Define exactly what those existing runs are, which repos/branches, where they outputs and restarts are on disk. Andrew listed the 12 supported experiments above, but there aren’t 12 data products right?

14

The main branches of the configs above in Andrew’s post don’t necessarily correspond to the control runs that we have. e.g. The 01deg_jra55_ryf config now uses JRA55 v1.4, but the control run available (at /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091/) uses JRA55 v1.3 (I guess this branch?). At the moment it’s not necessarily easy to match configs with control runs. Would be great if ACCESS-NRI could match these up, ensure the configs are correct and make some instructions for new users on where to find matching restarts / configs.

15

So we’ll need to know the names of the control run experiments, as they don’t necessarily correspond with the name of the repository, e.g. 01deg_jra55_ryf vs 01deg_jra55v13_ryf9091.

For each control run we’ll need

  1. experiment name
  2. experiment git repository URL (on GitHub) with full run commit history
  3. location of output files on gadi
  4. location of restart files on gadi (if different from outputs)
  5. Description of the experiment
  6. Ideally how it has been used: major studies/papers/experiments (this gives context as to it’s importance scientifically, and within the community, and means those without familiarity with COSIMA experiments can make choices about which experiment they use, as they may be able to utilise previous research in their own work)

@aekiss’s run summary tool is invaluable in collecting metadata on these runs. e.g. /g/data/hh5/tmp/cosima/access-om2-run-summaries/run_summary_01deg_jra55v13_ryf9091.csv but it does need some context and assistance in making sure all the information is readily available.

1 Like
16

I am regretting using the term “control run”. It is confusing with “control directory”. I should have used “reference run” or similar. Is there a standard term for these runs?

17

Should we start a shared document somewhere with these details?

18

Technically all the information is in the configuration files in the output directories (and the executable hashes), but I agree it’s not trivial to extract the configuration from this information. A traceable record of the github repository and commit hash might be useful to add here.

I’m happy to add the details of the 1-degree and 1/4-degree runs to these. @Aidan is there an access-hive/forum linked way to do this, or do we fallback on google sheets/docs? I feel like we used to have something like this, but I can’t find it…

“control run” sounds right to me. A single repository can result in multiple control runs.

19

Found it: ik11_outputs_directory - Google Sheets It’s long out of date.

20

That is convenient, but the visibility isn’t great, so I think probably best to just document on the forum, as it is useful information by itself. I thought about adding it to the ACCESS-OM2 wiki, and that is still an option, but decided in the end to make a new topic:

It is a wiki topic, so should be editable. If it isn’t let me know. I’ve tried to put in some background, but please do feel free to modify or add anything you think might be useful.

I’ve started with filling in the details for the 0.1 RYF run, and grabbed a description from the metadata.yaml file. That may be out of date now, not sure. It should be straightforward to copy that section for other runs and replace the content. Again, let me know if that is problematic, or you think it is not the way to go.