7th March 2025 - Introduction to Cosima-Cookbook/Cosima-Recipes & how to make it go - loading data fast

COSIMA 2025 training program
1

Friday 7th March 11AM AEDT, Zoom Link.

In this session, we’ll cover the basics of getting started with cosima-recipes, through to optimising loading datasets - to help perform analyses quicker & more effectively.

We’ll start with a brief history and overview of the cosima-cookbook and cosima-recipes, before we dive into tutorial notebooks to get up to speed with what cosima-recipes can help you with. Depending on the experience of the attendees, we’ll focus either on (or split into groups doing both):

  • Getting up to speed with the cosima-cookbook (0-60); or
  • Making it go: how to speed up loading your data, following the cosima cookbook approaches (60-100).

Pre-Requisites

ARE Setup

  • Module directories
    /g/data/xp65/public/modules
  • Modules
    conda/analysis3-25.02
  • Projects:
    gdata/xp65+gdata/$PROJECT
    +gdata/hh5+gdata/ik11+gdata/cj50 (these projects have most COSIMA model results)
    +gdata/oi10+gdata/ol01+gdata/fs38+gdata/p73 (these projects have more model results)
  • Compute Size of large or greater.

Screenshot 2025-02-20 at 9.38.48 am
Screenshot 2025-02-20 at 9.38.48 am1350×2402 351 KB

Slides

cookbook-tutorial.pdf (138.3 KB)

making_it_go.pdf (363.6 KB)

1 Like
2

Thanks for the great training session @CharlesTurner, @AndyHoggANU, et al.

I do recognise that many folks just want to get on with science and really don’t wan’t to deal with the details of how to make an “analysis ready” data object. I think there is much to be gained across the community - doing much more with given resources - by continuing the excellent efforts like today.

I’ll attach here a poster from OMO24 from myself and @ChrisC28 that was and is just about stimulating discussion on how the wider community can address reducing the cost in time and effort involved in getting past turning large groups of NetCDF files into analysis ready datasets for ocean, climate, and earth science.

Thanks again.

1 Like
3

Moore_OMO24_poster_29062024_Apaper.pdf (10.4 MB)

2 Likes
4

Thanks for coming along today everyone. Please fill in our feedback poll here, should take less than a minute….

5

Session feedback.

Anonymous

The beginners’ session was super-clear and easy to follow. The method of loading input {catalog = intake.cat.access_nri; ds = catalog[experiment].search(variable=variable).to_dask()} was really simple and useful From there, the tutorial explored slicing/plotting analyses which were helpful (more on resampling options would be good) An early exercise asked about chunking - it’d be good to explore this, too The advanced exercises included dask distributed computing - again, this would be a good follow-up Will definitely do the cartopy tute Thanks

Anonymous

I appreciated the format and content. It worked well for me and I found the information useful.

Anonymous

Great to be able to get everyone in one room with the experts to dive into chunking advice - very helpful

Anonymous

I really like the advices on chunking in the advanced session! Really helpful!

1 Like