CMIP7 Fast Track CMORisation — where we’re up to, and how to check status
Following the ACCESS-MOPPy v1.8.0 release announcement,
a status update on the CMIP7 Fast Track CMORisation itself: what has been produced, what has been checked, and how to see the state of any run for yourself.
There is a new progress dashboard, and a gate system behind it that changes how you should read “done”. The short version is at the top; the detail is below.
Dashboard: ACCESS CMIP7 Submission Tracker
What happens on the way out
MOPPy takes native ACCESS output — UM fields files, MOM and CICE history — and writes CMIP-compliant files directly, without the CMOR library. Four things happen as it does:
CMORisation — variable names, global attributes, DRS paths and file names, built on xarray and dask so the same code runs in a notebook or across hundreds of PBS jobs.
Physical range checks. Every file is checked against a per-variable physical envelope — 293 ACCESS-ESM1-6 variables with experiment-specific overrides, plus units, missing-value and finite-value checks. The bounds are deliberately broad: they exist to catch a unit, sign or conversion error, not to reject a plausible extreme. Every rule is published and filterable in the QC ranges reference,
and you can print them from the command line:
moppy-qc --show-ranges --variable tas
WCRP compliance. We run the CF suite (cf:1.11) and the WCRP CMIP suite (wcrp_cmip7:1.0, backed by esgvoc) over the output — metadata, controlled-vocabulary values, DRS path and file name. See compliance testing.
QC diagnostic plots, for scientific review: a spatial snapshot and a global-mean timeseries with min/max shading per file, with the published ACCESS-ESM1-5 CMIP6 series overlaid where available, so drift against the previous submission is visible at a glance. See
QC validation.
On top of that, files are repacked for cloud access.
The dashboard, and the release gates
ACCESS CMIP7 Submission Tracker
Each CMORised variable is treated as a candidate for publication rather than a finished
product. The batch report records the outcome of three release gates against every task:
| Gate | What it checks |
|---|---|
R range |
the physical-range check, run on every file |
W wcrp |
the CF + WCRP CMIP compliance suites |
K repack |
the cloud repacking |
Each gate records pass, warn or fail together with the evidence behind it — or
not run where the check has yet to be done, and implied where it can be inferred but was not recorded. A gate counts as cleared when it passed, passed with a warning, or was implied; clearing all three is what makes a variable ready for review.
Where a variable writes several files the worst outcome is kept, so a single bad split is never averaged away. That means you can see what was actually verified, instead of inferring it from a job not having failed.
Background: release gates.
A GitHub Action re-checks the CMORisation status and updates the dashboard nightly.
Read grey as “not yet checked”, not “failed”
Everywhere else range and wcrp read not run, and repack reads implied rather than verified. The gate results come out of the reruns described below, so coverage fills in as those progress. If you open the dashboard today and see a wall of grey tiles, nothing has failed — the checks have not reached those runs yet.
A short walkthrough of the dashboard
Slides covering how to read the two legends, and how to get from the overview down to the state of a single run:
moppy_dashboard_walkthrough.pdf (1.5 MB)
It follows one run end to end — ACCESS-ESM1.6 / esm-hist / r29i1p1f1, one of the members that does have a complete set of gates.
Where the data is
The current CMORised outputs are in /scratch/p73/ESM1p6_CMORised. They are progressively being rerun with the latest version of MOPPy, since several requirements flagged by the compliance checkers have been adjusted.
Two things to know:
- Write access is limited to the MOPPy team, so please come to us rather than trying to write into those directories.
- The output needs to stay in those folders. The nightly dashboard update reads them
directly — moving the files will make the dashboard go stale or wrong.
We have also run ESMValTool recipes and ILAMB over the ensembles.
A caveat on the compliance checker
The WCRP checker raises a lot of false positives at level 2 (non-blocking). We have reached out to the maintainers, but this looks like something to navigate rather than something that gets fixed in time. It should be fine in practice, as long as level-2 findings are triaged rather than treated as blocking.
What’s next
The focus so far has been the baseline Fast Track monthly variables. Daily and sub-daily data required for Fast Track is being worked on now.
One thing to expect: the CMIP7 controlled vocabulary is updated regularly, and each update needs a corresponding MOPPy update. These are usually minor and should not hold up publication.
Known issues, and reporting new ones
If you hit something, please check the issue tracker first — it may already be flagged. Known items include the basin file and the zostoga calculation, among others.
Please do report anything you find. We need MOPPy to stay fit for purpose, and issues we don’t hear about don’t get fixed. Reply on this thread, or open an issue.
Tagging @RhaegarZeng and @peterdobro