COSIMA Hackathon - G1 training

[dougiesquire (Dougie Squire, ACCESS-NRI Ocean/Sea-Ice Modelling Team)]

COSIMA hackathon v4.0 - G1 training

This is material to support training for the G1 group at the COSIMA hackathon v4.0, 1st July 2024.

It is aimed at people who are new to COSIMA recipes. After completing this training, users will be able to use and modify COSIMA recipe notebooks using the NCI Australian Research Environment (ARE).

Prerequisites

Please make sure you’ve completed the steps in this topic post prior to the hackathon.

1. Set up SSH keys for accessing Gadi and Github

First follow these instructions on the ACCESS-Hive to set up an SSH key to access Gadi from your local machine.

Now we’ll set up an SSH key to push/pull from Github from our local machine and use SSH agent forwarding to allow our local SSH keys to also be used on Gadi.

1. Follow the Github documentation to generate a new SSH key for Github and add the SSH key to the ssh-agent.

2. Follow the Github documentation to add the SSH key to your Github account.

3. Follow the Github documentation to test your SSH connection from your local machine

4. Set up SSH forwarding from your local machine to Gadi. Using your favourite text editor, open ~/.ssh/config and add the following line to the gadi host that you set up earlier:

   Host gadi
     ForwardAgent yes
   

5. Test that the SSH agent forwarding was successful:

   $ ssh gadi
   

   then

   $ ssh -T git@github.com
   

2. Starting an ARE JupyterLab session

There are lots of existing resources on how to use the NCI Australian Research Environment (ARE). The instructions on the ACCESS-Hive and the NCI ARE User Guide include lots of useful information.

Steps:

1. Log in to ARE using your NCI username and password

2. Select the JupyterLab app and enter the following configuration settings (The fields marked with * are found in “Advanced Options”):

   Field	Entry
   Walltime	2 hours
   Queue	normalbw
   Compute Size	Medium
   Project	Choose one
   Storage	gdata/hh5+gdata/ik11+gdata/cj50
   Module directories*	/g/data/hh5/public/modules
   Modules*	conda/analysis3-unstable

   Note, the storage section must contain all the /g/data projects you need access to. Above is only a minimum set needed for COSIMA recipes. Note also that we are using the shared analysis3-unstable conda environment from hh5 project. This contains all the Python packages we need to run COSIMA recipes.

3. Click the “Launch” button. You should see something like this:

   

   You may have to wait a few minutes for your queued job to start running. When your JupyterLab session is running, you should see something like this:

   

   Click the “Open JupyterLab” button to open your JupyterLab session in a new window.

3. Cloning COSIMA recipes

COSIMA recipes is a collection of Jupyter notebooks for analysing and plotting ocean model output. Most recipes make heavy use of the COSIMA cookbook which is a Python package for searching and loading model output on Gadi. Note the COSIMA cookbook is now largely unsupported. ACCESS-NRI have built the ACCESS-NRI Intake catalog, which provides similar functionality to the COSIMA cookbook.

COSIMA recipes is a repository on Github. In order to use the recipe notebooks on the ARE we need to create a copy of COSIMA recipes on Gadi. In git terminology, this is called “cloning” the repository.

We’re going to do this from the terminal available in our ARE Jupyterlab session, but you could equally do it from your own terminal after SSHing into Gadi.

1. Go to your Jupyterlab window, click on the folder icon at the top left, click on the blue “+”, scroll down and click on the “Terminal” icon to open a terminal window:

   

2. From the terminal window, navigate to where you would like to clone the COSIMA recipes. E.g.:

   $ mkdir -p /scratch/ik11/$USER/cosima_hackathon_2024
   $ cd /scratch/ik11/$USER/cosima_hackathon_2024
   

   Note, files in /scratch will be automatically removed when they are older than 100 days (see details here). Use a project directory on /g/data/ for something more permanent.

3. Clone the COSIMA recipes Github repo:

   $ git clone git@github.com:COSIMA/cosima-recipes.git
   

4. Navigate to a recipe notebook and open it in JupyterLab

We’ll be running the COSIMA_Cookbook_Tutorial.ipynb notebook. Click on the folder icon at the top left of your JupyterLab session and user the interface to navigate to your clone of the COSIMA recipes.

Within the Tutorials directory, double click COSIMA_Cookbook_Tutorial.ipynb to open the notebook.

You can execute cells in the notebook using the arrow button in the control panel along the top of the notebook or using <shift>+<enter>

5. Self-led tutorials

That concludes the guided segment of the G1 training. For the afternoon of the Hackathon, it’s recommended that you either join a group working on an issue, or have a go at one of the following tutorials. We’ll be around to help with questions!

• COSIMA workshop 2023 training: clone this repo in the same way as the COSIMA recipes and run the notebooks
• Xarray in 45 mins tutorial
• ACCESS-NRI Intake catalog quickstart tutorial
• Git tutorial from Monash School of Mathematics
• Github’s introductory material for git and Github
• Xarray, GitHub, and many other tutorials from Project Pythia Foundations: a community developed resource that contains several self-led tutorials, specifically for those in the climate and geo sciences
• Go through some other COSIMA recipes!

[dougiesquire (Dougie Squire, ACCESS-NRI Ocean/Sea-Ice Modelling Team)]

COSIMA hackathon v5.0 - G1 training

This is material to support training for the G1 group at the COSIMA hackathon v5.0, 8th Sept 2025. It is an update to the G1 training from last year’s COSIMA hackathon.

It is aimed at people who are new to COSIMA recipes. After completing this training, users will be able to use and modify COSIMA recipe notebooks using the NCI Australian Research Environment (ARE).

Prerequisites

Please make sure you’ve completed the steps in this topic post prior to the hackathon.

1. Logging in to the ARE

The Australian Research Environment is a web-based graphical interface for performing your computational research on Gadi, provided by NCI. There are lots of existing resources on how to use the NCI ARE. The instructions on the ACCESS-Hive and the NCI ARE User Guide include lots of useful information.

In this training, we’ll use the ARE to run both terminal commands and a JupyterLab instance on Gadi nodes. The terminal commands run below could also be run from your own terminal after SSHing into Gadi.

To log in to the ARE, go to https://are.nci.org.au/ and enter your NCI username and password.

2. Set up Github authentication

In order to interact with Github from Gadi we need to authenticate to Github. Last year, we manually set up SSH keys to do this. This year we’ll use the Github command-line tool gh to greatly simplify the process.

Steps:

1. From your ARE dashboard, select the “Gadi Terminal” app. This will open a terminal with SSH connection to a Gadi login node.
2. In the terminal window enter:

   $ module use /g/data/vk83/modules
   $ module load gh
   $ gh auth login
   

   You will be prompted with a few questions. Answer as follows:

   ? What account do you want to log into? GitHub.com
   ? What is your preferred protocol for Git operations on this host? HTTPS
   ? Authenticate Git with your GitHub credentials? Y
   ? How would you like to authenticate GitHub CLI? Login with a web browser
   ! First copy your one-time code: ABCD-1234
   Press Enter to open github.com in your browser... 
   ! Failed opening a web browser at https://github.com/login/device
     exec: "xdg-open,x-www-browser,www-browser,wslview": executable file not found in $PATH
     Please try entering the URL in your browser manually
   ✓ Authentication complete.
   

   As you can see, gh will fail to automatically open the web browser from Gadi so we’ll do this manually. In a new browser tab, go to https://github.com/login/device and follow the prompts. You will be asked for a verification code, which is printed to your terminal (e.g. ABCD-1234 above). Then click “Authorize GitHub”.
3. That’s it. You do not need to do this again on Gadi.

3. Cloning COSIMA recipes

COSIMA recipes is a collection of Python notebooks for analysing and plotting ocean model output.

COSIMA recipes is a repository on Github. In order to use the recipe notebooks on the ARE we need to create a copy of COSIMA recipes on Gadi. In git terminology, this is called “cloning” the repository.

Steps:

1. From your ARE terminal window, navigate to where you would like to clone the COSIMA recipes. E.g.:

   $ mkdir -p /scratch/ik11/$USER/cosima_hackathon_2025
   $ cd /scratch/ik11/$USER/cosima_hackathon_2025
   

   Note, files in /scratch will be automatically removed when they are older than 100 days (see details here). Use a project directory on /g/data/ for something more permanent.

2. Clone the COSIMA recipes Github repo:

   $ git clone https://github.com/COSIMA/cosima-recipes.git
   

   You will now have a directory called cosima-recipes containing all of the COSIMA recipes.

4. Starting an ARE JupyterLab session

JupyterLab is a web-based interactive development environment for Python, R and Julia notebooks. We’ll use JupyterLab to run and modify the COSIMA recipes Python notebooks.

Steps:

1. From your ARE dashboard, select the “JupyterLab” app and enter the following configuration settings (The fields marked with * are found in “Advanced Options”):

   Field	Entry
   Walltime	2 hours
   Queue	normalbw
   Compute Size	Medium
   Project	Choose one
   Storage	gdata/xp65+gdata/ik11+gdata/cj50
   Module directories*	/g/data/xp65/public/modules
   Modules*	conda/analysis3

   Note, the storage section must contain all the /g/data projects you need access to. Above is only a minimum set needed for most COSIMA recipes. Note also that we are using the shared analysis3 conda environment from the xp65 project. This contains all the Python packages we need to run COSIMA recipes.

2. Click the “Launch” button. You should see something like this:

   

   You may have to wait a few minutes for your queued job to start running. When your JupyterLab session is running, you should see something like this:

   

   Click the “Open JupyterLab” button to open your JupyterLab session in a new window.

5. Navigate to a recipe notebook and open it in JupyterLab

We’ll be running the COSIMA_Cookbook_Tutorial.ipynb notebook. Click on the folder icon at the top left of your JupyterLab session and user the interface to navigate to your clone of the COSIMA recipes.

Within the Cooking-Lessons-101-Tutorials directory, double click COSIMA_Cookbook_Tutorial.ipynb to open the notebook.

You can execute cells in the notebook using the arrow button in the control panel along the top of the notebook or using <shift>+<enter>

5. Self-led tutorials

That concludes the guided segment of the G1 training. For the rest of the Hackathon, it’s recommended that you either join a group working on an issue, or go through the following COSIMA recipes tutorials in the Cooking-Lessons-101-Tutorials directory. We’ll be around to help with questions!

1. ACCESS-NRI_Intake_Catalog.ipynb
2. Making_Maps_with_Cartopy.ipynb

You may also find these additional resources/tutorials useful:

• Xarray in 45 mins tutorial
• ACCESS-NRI workshop 2024 training on finding and analysing large data
• ACCESS-NRI workshop 2024 training on advanced Git
• COSIMA workshop 2023 training
• ACCESS-NRI Intake catalog quickstart tutorial
• Git tutorial from Monash School of Mathematics
• Github’s introductory material for git and Github
• Xarray, GitHub, and many other tutorials from Project Pythia Foundations: a community developed resource that contains several self-led tutorials, specifically for those in the climate and geo sciences