# Introduction to COSIMA Cookbook, Data Explorer and ACCESS-NRI Intake Catalog **URL:** \ **Category:** Workshops\ **Tags:** cosima, cosima-cookbook, training, cosima-workshop-2023\ **Created:** [26 August 2023 08:06 UTC](https://forum.access-hive.org.au/t/introduction-to-cosima-cookbook-data-explorer-and-access-nri-intake-catalog/1144 "2023-08-26T08:06:34Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [26 August 2023 08:06 UTC](https://forum.access-hive.org.au/t/introduction-to-cosima-cookbook-data-explorer-and-access-nri-intake-catalog/1144/1 "2023-08-26T08:06:35Z")
# About This is material to support a training session on the [COSIMA Cookbook](https://github.com/COSIMA/cosima-cookbook) using its Data Explorer component. The [ACCESS-NRI Intake catalog](https://forum.access-hive.org.au/c/access-workshop-2023/61) will also be introduced to show similar and complementary functionality. This training is aimed at people who have never used (or have forgotten how to use) the [COSIMA Recipes](https://github.com/COSIMA/cosima-recipes) for accessing and analysing COSIMA model output. The training will utilise the [NCI ARE](https://opus.nci.org.au/display/Help/ARE+User+Guide) (Australian Research Environment) [JupyterLab App](https://opus.nci.org.au/display/Help/3.+JupyterLab+App). This is user-friendly way to run Jupyter Notebooks at NCI, which makes it possible to access the data required for this training session. # Who is this for? Attendees of the COSIMA Workshop, a satellite meeting of the [ACCESS 2023 Community Workshop](https://forum.access-hive.org.au/c/access-workshop-2023/61). Attendees will need some familiarity with the python programming language. # Preparation Before Training Day There are three prerequisites, two to access NCI resources, and one to get the jupyter python notebooks. _All_ perequisites need to be completed **WELL BEFORE** the day of training. We only have an hour for the training, and there will not be time to do administration _and_ follow the training. ## NCI Account You _must_ have an NCI account linked to a project with remaining compute allocation. This training will require submitting a job to the NCI batch queuing system. Without remaining compute allocation your submitted job will never run. If you aren’t sure which project to use, ask your supervisor, colleagues or relevant local IT support. If you do not have an NCI account follow the NCI [instructions to create one](https://opus.nci.org.au/display/Help/Setting+up+your+NCI+Account). ## NCI Projects To access the data you need to complete the tutorial you should make sure you are also a member of the following projects at NCI: - `hh5` - `ik11` - `cj50` - `xp65` - `p73` - `oi10` - `jk72` - `fs38` You can see the projects you are a member of at [Log in | MyNCI](https://my.nci.org.au/mancini). Or run this command on `gadi` ```auto /scratch/public/COSIMA_tutorial/check_projects ``` ## Get Notebooks The training uses two notebooks in a `git` repository. You will need to download these notebooks to an NCI filesystem that is accessible to the ARE session you just started. The recommended location is your home directory on `gadi`, as this is always accessible to an ARE JupyterLab session. To do this log in to `gadi` and copy and paste this command: ```auto git clone git@github.com:ACCESS-Hive/cosima-training-workshop-2023.git ``` This will have created a directory called `cosima_training` in your gadi home directory. # Training Session ## Start ARE JupyterLab Session This will be the first thing we do at the training session, but feel free to test this out before the training day to check it is all working correctly for you. The [instructions from the ACCESS-NRI Intake catalogue docs](https://access-nri-intake-catalog.readthedocs.io/en/latest/usage/how.html#using-the-catalog-on-the-are) are excellent and worth reading for background, and of course the [NCI ARE Documentation](https://opus.nci.org.au/display/Help/ARE+User+Guide) is also available. Steps: 1. Log in to ARE using your NCI username and password: [https://are.nci.org.au](https://are.nci.org.au) ![ARE](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/38691708aadd177bc3929710588bc1ed23907107.png) 1. Select the JupyterLab app ![icon](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/d459475f86a3914789ea7eed0e3595786604b373.png) For this training here are some recommended settings for your ARE session that have been tested and will work: | | | | --- | --- | | Walltime | 2 hours | | Queue | normalbw | | Compute Size | Medium | | Project | _choose one_ | | Storage | _see below_ | You must _choose one_ of the projects you have available to you. This must be a project with compute allocation remaining as discussed above in prerequisites. The storage section _must_ contain all the `/g/data` projects you need to access. Use the following to include all the projects specified above: ```auto gdata/hh5+gdata/ik11+gdata/cj50+gdata/xp65+gdata/p73+gdata/oi10+gdata/dk92+gdata/fs38+scratch/public ``` Note: you can hover over the box above, and click on the icon in the top right corner to copy it into your buffer and then paste it into the ARE window. Click on “Advanced Settings” and set the following options **Module directories** ```auto /g/data/hh5/public/modules ``` **Modules** ```auto conda/analysis3-unstable ``` Note: can cut and paste these using the shortcut mentioned above. Push the submit button and then you will have a window that looks something like this: ![ARE_queue](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/0c4be97188937a9867b9917ea4ea143f212d0e2c.png) You may wait a few minutes for your queued job to start running, which is where your JupyterLab application runs. When your JupyterLab session is ready the screen will change, an “Open JupyterLab” button. Push it and your JupyterLab session will open in a new window. ![JupyterLab_ready](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/b2aa0d0e4390936681865bcb3930dd693c564ee1.jpeg) ## Find Tutorial Notebooks In the JupyterLab window use the file browser to navigate to the directory where you downloaded the jupyter notebook files in the preparation section above. Double-click on the `home` directory in the file browser: ![jupyterlab_filebrowser](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/39ac6a14fa5b588a322e7f327839fa2dcf193ed6.png) and then double-click on the `cosima-training-workshop-2023` directory. ## Notebook 1: Finding COSIMA Data The [first notebook](https://github.com/ACCESS-Hive/cosima-training-workshop-2023/blob/main/Finding_COSIMA_data.ipynb) covers using the explorer tools in the COSIMA Cookbook database to find data. ## Notebook 2: Comparing Sea Surface Height Data In the [second notebook](https://github.com/ACCESS-Hive/cosima-training-workshop-2023/blob/main/Sea_level.ipynb) you’ll use what you’ve learned from the first part to load sea surface height data, do some calculations and plot the result. ## Notebook 3: Introducing the ACCESS-NRI Intake Catalog If there is time, or for those who are particularly interested, the [third notebook](https://github.com/ACCESS-Hive/cosima-training-workshop-2023/blob/main/Intake.ipynb) is a very short introduction to the [ACCESS-NRI Intake Catalogue](https://access-nri-intake-catalog.readthedocs.io/en/latest/). The catalog was developed by @dougiesquire. The aim is to show similarities and how the cookbook and intake catalog can be complementary: how the same load operation is done with the ACCESS-NRI Intake catalog, and what other datasets are available via the Intake catalog. # Clean-up When finished make sure you save your work, close the tab and then Delete your running JupyterLab app, otherwise it will continue to consume compute resources and eventually stop when it reaches the walltime limit. You are free to keep the notebooks you’ve been working on for reference. If you no longer want them you can use ```bash rm -rf cosima-training-workshop-2023 ``` but BE CAREFUL with `rm -rf`. This will happily recursively delete your entire directory tree if given the wrong arguments (or right, if that is what you want to do). # Conclusion Hopefully you now know how to find COSIMA data, load it and do some meaningful analysis. Also you can now use the explorer to locate data, and learn the syntax for loading data in your own notebooks. If there was time maybe you’ve learnt a bit how the intake catalogue works, how it is different to the COSIMA Cookbook, and complimentary. ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [31 August 2023 05:31 UTC](https://forum.access-hive.org.au/t/introduction-to-cosima-cookbook-data-explorer-and-access-nri-intake-catalog/1144/2 "2023-08-31T05:31:24Z")
Contact @rogeredberg if you are unable to set up an NCI account from your institution, or if you don’t have an NCI compute project you can use. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [1 September 2023 03:54 UTC](https://forum.access-hive.org.au/t/introduction-to-cosima-cookbook-data-explorer-and-access-nri-intake-catalog/1144/3 "2023-09-01T03:54:49Z")
Thanks heaps to @hrsdawson for volunteering to assist. @dougiesquire can make the last 20 in case we do make it to the intake catalogue (thanks @dougiesquire). Anyone else able to come and assist, particularly for the first 10-15 minutes go get everyone up and running?