COSIMA Hackathon v4.0 - G1 training

COSIMA Workshops
, , , ,
1

About

This is material to support training for the G1 group at the COSIMA hackathon v4.0, 1st July 2024.

It is aimed at people who are new to COSIMA recipes. After completing this training, users will be able to use and modify COSIMA recipe notebooks using the NCI Australian Research Environment (ARE).

Prerequisites

Please make sure you’ve completed the steps in this topic post prior to the hackathon.

1. Set up SSH keys for accessing Gadi and Github

First follow these instructions on the ACCESS-Hive to set up an SSH key to access Gadi from your local machine.

Now we’ll set up an SSH key to push/pull from Github from our local machine and use SSH agent forwarding to allow our local SSH keys to also be used on Gadi.

  1. Follow the Github documentation to generate a new SSH key for Github and add the SSH key to the ssh-agent.

  2. Follow the Github documentation to add the SSH key to your Github account.

  3. Follow the Github documentation to test your SSH connection from your local machine

  4. Set up SSH forwarding from your local machine to Gadi. Using your favourite text editor, open ~/.ssh/config and add the following line to the gadi host that you set up earlier:

    Host gadi
  ForwardAgent yes

  5. Test that the SSH agent forwarding was successful:

    $ ssh gadi

    then

    $ ssh -T git@github.com

2. Starting an ARE JupyterLab session

There are lots of existing resources on how to use the NCI Australian Research Environment (ARE). The instructions on the ACCESS-Hive and the NCI ARE User Guide include lots of useful information.

Steps:

  1. Log in to ARE using your NCI username and password

  2. Select the JupyterLab app and enter the following configuration settings (The fields marked with * are found in “Advanced Options”):

    Field Entry
    Walltime 2 hours
    Queue normalbw
    Compute Size Medium
    Project Choose one
    Storage gdata/hh5+gdata/ik11+gdata/cj50
    Module directories* /g/data/hh5/public/modules
    Modules* conda/analysis3-unstable

    Note, the storage section must contain all the /g/data projects you need access to. Above is only a minimum set needed for COSIMA recipes. Note also that we are using the shared analysis3-unstable conda environment from hh5 project. This contains all the Python packages we need to run COSIMA recipes.

  3. Click the “Launch” button. You should see something like this:

    Screenshot 2024-06-27 at 4.22.07 PM
    Screenshot 2024-06-27 at 4.22.07 PM1868×672 49.2 KB

    You may have to wait a few minutes for your queued job to start running. When your JupyterLab session is running, you should see something like this:

    Screenshot 2024-06-27 at 4.25.46 PM
    Screenshot 2024-06-27 at 4.25.46 PM1874×774 64.1 KB

    Click the “Open JupyterLab” button to open your JupyterLab session in a new window.

3. Cloning COSIMA recipes

COSIMA recipes is a collection of Jupyter notebooks for analysing and plotting ocean model output. Most recipes make heavy use of the COSIMA cookbook which is a Python package for searching and loading model output on Gadi. Note the COSIMA cookbook is now largely unsupported. ACCESS-NRI have built the ACCESS-NRI Intake catalog, which provides similar functionality to the COSIMA cookbook.

COSIMA recipes is a repository on Github. In order to use the recipe notebooks on the ARE we need to create a copy of COSIMA recipes on Gadi. In git terminology, this is called “cloning” the repository.

We’re going to do this from the terminal available in our ARE Jupyterlab session, but you could equally do it from your own terminal after SSHing into Gadi.

  1. Go to your Jupyterlab window, click on the folder icon at the top left, click on the blue “+”, scroll down and click on the “Terminal” icon to open a terminal window:

    Screenshot 2024-06-27 at 4.38.16 PM
    Screenshot 2024-06-27 at 4.38.16 PM2042×756 71.6 KB

  2. From the terminal window, navigate to where you would like to clone the COSIMA recipes. E.g.:

    $ mkdir -p /scratch/ik11/$USER/cosima_hackathon_2024
$ cd /scratch/ik11/$USER/cosima_hackathon_2024

    Note, files in /scratch will be automatically removed when they are older than 100 days (see details here). Use a project directory on /g/data/ for something more permanent.

  3. Clone the COSIMA recipes Github repo:

    $ git clone git@github.com:COSIMA/cosima-recipes.git

4. Navigate to a recipe notebook and open it in JupyterLab

We’ll be running the COSIMA_Cookbook_Tutorial.ipynb notebook. Click on the folder icon at the top left of your JupyterLab session and user the interface to navigate to your clone of the COSIMA recipes.

Within the Tutorials directory, double click COSIMA_Cookbook_Tutorial.ipynb to open the notebook.

You can execute cells in the notebook using the arrow button in the control panel along the top of the notebook or using <shift>+<enter>

5. Self-led tutorials

That concludes the guided segment of the G1 training. For the afternoon of the Hackathon, it’s recommended that you either join a group working on an issue, or have a go at one of the following tutorials. We’ll be around to help with questions!

4 Likes