ACCESS-MOPPy v1.8.0 — our first stable release!

ACCESS-NRI Releases
, , ,
1

:tada: ACCESS-MOPPy v1.8.0 — our first stable release!

MOPPy Logo
MOPPy Logo707×848 331 KB

We’re excited to announce ACCESS-MOPPy v1.8.0, the first release out of beta! After many beta iterations, the tool is now stable enough for general use across the ACCESS community.

ACCESS-MOPPy (Model Output Post-Processor in Python) is a CMORisation tool that post-processes raw ACCESS model output and produces CMIP-compliant datasets — the same job APP4 used to do, but written in Python, with a modern Python API, Dask-powered parallelism, and a batch-processing system built for NCI Gadi.

:movie_camera: Start here: a 10 minute walkthrough

CMORising ACCESS output for CMIP7 FastTrack — a quick start with ACCESS-MOPPy

It runs through the whole Gadi workflow end to end: loading the environment, writing a config, submitting the batch, and watching it run. If you only have ten minutes, spend them here — it follows the Quick start guide step for step.

(The narration voice is synthetic and the script was drafted with AI assistance from our documentation. The workflow it shows is the real, maintained one.)

Quick start — the four steps

Every production run is the same four steps, whatever project you’re CMORising for:

1. load the analysis3 environment
2. write a batch config file        (which variables + where the data is)
3. moppy-cmorise my_config.yml      (submits one PBS job per variable)
4. moppy-tui --db .../cmor_tasks.db (watch it run)

The Quick start walks through those with the smallest config that actually works, then hands you the metadata block to paste in for your project:

What it does

  • CMORises ACCESS model output (ACCESS-ESM1-5, ACCESS-ESM1-6, ACCESS-CM3 (partial), ACCESS-OM3 (partial)) into CMIP-compliant NetCDF, following the CMIP6, CMIP6Plus, and CMIP7 controlled vocabularies and CMOR tables.
  • Python API for interactive, notebook-driven CMORisation of individual variables.
  • Batch processing system for PBS/HPC environments — submit hundreds of variables as parallel, self-tracking PBS jobs from a single YAML config.
  • Dask-enabled for scalable, parallel, out-of-core computation.
  • Built-in QC checks and diagnostic plots for output validation.
  • Support for both the classic CMIP6 table.variable naming (e.g. Amon.pr) and the new CMIP7 full branded-name convention (e.g. atmos.rsds.tavg-u-hxy-u.mon.GLB).
  • Not limited to Gadi or ACCESS-NRI infrastructure — it will run anywhere with the right Python environment.

Using it in a notebook

from access_moppy import ACCESS_ESM_CMORiser

cmoriser = ACCESS_ESM_CMORiser(
    input_folder="/g/data/p73/archive/CMIP7/ACCESS-ESM1-6/historical/MyRun",
    compound_name="Amon.pr",       # table.variable format (CMIP6/6Plus) or full branded name (CMIP7)
    experiment_id="historical",
    source_id="ACCESS-ESM1-6",
    variant_label="r1i1p1f1",
    grid_label="gn",
    activity_id="CMIP",
    cmip_version="CMIP6",          # or "CMIP6Plus" / "CMIP7"
)

cmoriser.run()                     # run the CMORisation (Dask-backed)
cmoriser.variable_mapping          # inspect the variable mapping (rich display in Jupyter)
ds = cmoriser.to_dataset()         # inspect the result as an xarray Dataset
cmoriser.write()                   # write CMIP-compliant NetCDF, split by CMIP time-chunk convention

ACCESS_ESM_CMORiser can either auto-discover input files from a payu archive root (input_folder), or take an explicit file list (input_data) if you need finer control. See the Getting Started tutorial and the notebooks in notebooks/ for worked examples, including ENSO recipes, ILAMB variable CMORisation, and ESMValTool integration.

Batch processing on Gadi

For production-scale runs — many variables, full experiments — use the PBS batch system:

  1. Load the environment:

    module use /g/data/xp65/public/modules
module load conda/analysis3

  2. Write a cmip7_test.yml — this is a CMIP7 FastTrack run, using the branded variable names:

    variables:
  - atmos.tas.tavg-h2m-hxy-u.mon.GLB
  - atmos.pr.tavg-u-hxy-u.mon.GLB
  - ocean.tos.tavg-u-hxy-sea.mon.GLB

cmip_version: CMIP7
experiment_id: historical
source_id: ACCESS-ESM1-6
variant_label: r1i1p1f1
activity_id: CMIP

# Describes what this run branched from. Required for publication;
# set branch_time_in_parent to the real branch day for your run.
parent_info:
  parent_experiment_id: piControl
  parent_activity_id: CMIP
  parent_source_id: ACCESS-ESM1-6
  parent_variant_label: r1i1p1f1
  parent_time_units: "days since 0001-01-01 00:00:00"
  parent_mip_era: CMIP7
  branch_time_in_child: 0.0
  branch_time_in_parent: 0.0
  branch_method: standard

input_folder: "/g/data/<project>/archive/CMIP7/ACCESS-ESM1-6/historical/<run>"
output_folder: "/scratch/<project>/<user>/cmor/cmip7_test"

# Keep the first run short while you check the output.
start_year: 1850
end_year: 1850

queue: "normalbw"
cpus_per_node: 12
mem: "64GB"
jobfs: 100GB
walltime: "06:00:00"
scheduler_options: "#PBS -P <project>"
storage: "gdata/p73+gdata/xp65+scratch/<project>"

worker_init: |
  module use /g/data/xp65/public/modules
  module load conda/analysis3

    Prefer the classic CMIP6 names? Set cmip_version: CMIP6 (or CMIP6Plus) and list variables as Amon.tas, Amon.pr, Omon.tos instead. Branded and compound names are interchangeable as input, and matching is case-insensitive.

  3. Submit — this submits a monitor job and returns immediately, so it’s safe to log out:

    moppy-cmorise cmip7_test.yml

  4. Monitor progress with any of:

    moppy-tui --db <output_folder>/cmor_tasks.db   # terminal dashboard
moppy-dashboard                                 # Streamlit web dashboard (localhost:8501)
moppy-batch-report --db <output_folder>/cmor_tasks.db   # JSON/summary report

Behind the scenes, moppy-cmorise submits one PBS job per variable, tracks everything in a SQLite database, and automatically retries jobs that fail or die without reporting (e.g. OOM) — a dropped SSH connection or a laptop going to sleep won’t stop a run in progress. Full configuration reference (resources, jobfs staging, QC plots, DRS output structure, etc.) is in the Batch Processing guide.

Documentation

Full documentation, including installation, tutorials, how-to guides, and API reference, is at:

:open_book: https://access-moppy.readthedocs.io

Some good starting points:

Source code, issue tracker, and releases: GitHub - ACCESS-NRI/ACCESS-MOPPy: ACCESS-MOPPy Admins: @rbeucher, @rhaegar325 · GitHub

:movie_camera: More training videos — coming soon

The quick start walkthrough above is the first of a series of short training videos we’re putting together, covering interactive CMORisation in notebooks, setting up and submitting batch jobs on Gadi, monitoring and troubleshooting runs, and working with CMIP7 FastTrack. Details on timing and where to find them will be posted here on the Forum as they’re ready — stay tuned!

Feedback and support

This is our first stable release, so please help us harden it further:

  • Bugs / feature requests: GitHub Issues
  • Questions: reply on this thread, or contact ACCESS-NRI support
  • Contributions: PRs welcome!

Thanks to everyone who tested the beta releases and reported issues — this release wouldn’t be here without you.

1 Like
CMIP7 Fast Track CMORisation — where we’re up to, and how to check status