21st Feb 2025 - Building and Using Intake Datastores

COSIMA 2025 training program
1

Friday 21st Feb on zoom

The goal of this session is to demonstrate how to build and use an intake datastore for individual experiments. This is targeted at analysis of ongoing experiments, or work related to a individual project. For analysis of shared and community model experiments (and some observational data) - use the access-nri-intake-catalog instead.

ACCESS-NRI Intake Docs:

https://access-nri-intake-catalog.readthedocs.io/en/latest/datastores/quickstart.html

To use the latest builders for making datastores, don’t use the normal hh5 conda/analysis environments. Use these settings in ARE:

ARE set-up:

This conda environment is still in development, report issues at GitHub - ACCESS-NRI/ACCESS-Analysis-Conda: A repository for the squashfs'd MED conda environments

  • Module directories
    /g/data/xp65/public/modules
  • Modules
    conda/analysis3-25.02
  • Projects:
    gdata/xp65+gdata/$PROJECT
    +gdata/hh5+gdata/ik11+gdata/cj50 (these projects have most COSIMA model results)
    +gdata/oi10+gdata/ol01+gdata/fs38+gdata/p73 (these projects have more model results)
  • Compute Size of large or greater.

Exercise:
GitHub - ACCESS-NRI/intake-training: A collection of training material on Intake

Example COSIMA recipe to run with your datastore, e.g:
https://github.com/COSIMA/cosima-recipes/blob/main/Recipes/Compare_SSH_model_obs.ipynb

1 Like
2

Session Notes

Presenters: @CharlesTurner @anton

  • Overall structure - 20 mins presentation, 20 mins working through exercises, followed by open discussion.
  • This session is to demonstrate how to build your own intake datastores.
  • Intake catalog is used for searching and discovering the existing output after running the ACCESS models.
  • Ongoing work:
    • Panantarctic (GFDL-OM4) results are in xp65 envs, available for testing.
    • Request additional datasets in the main catalog.
    • NCI also have intake catalog.

Important Links

  1. Training materials - 21st Feb 2025 - Building and Using Intake Datastores
  2. Training exercises GitHub repository - GitHub - ACCESS-NRI/intake-training: A collection of training material on Intake
  3. ACCESS-NRI Intake Catalog Docs - Quickstart tutorial — ACCESS-NRI Intake catalog documentation
  4. ACCESS-Hive Docs - https://access-hive.org.au
  5. ACCESS-Hive Forum - https://forum.access-hive.org.au
  6. Raise an issue here to have your experiment added to intake catalogue (use “Catalog data request” template): GitHub · Where software is built
2 Likes