# Tennessee Leeuwenburg: Auto-encoder training sessions **URL:** \ **Category:** Atmosphere/ML 2026 Training/Discussion Sessions\ **Created:** [12 August 2026 04:46 UTC](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788 "2026-08-12T04:46:02Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![NBateman-bit](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/nbateman-bit/32/45_2.png) [@NBateman-bit](https://forum.access-hive.org.au/u/NBateman-bit)\ **Post date:** [12 August 2026 04:46 UTC](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/1 "2026-08-12T04:46:02Z")
**Auto-encoder training sessions (Day 1 13:30-15:00 & Day 2 13:30-15:00)** Facilitated by: Tennessee Leeuwenburg (Bureau of Meteorology) This split-session tutorial provides an on-ramp to understanding and training machine learning models, through the example of autoencoders. Autoencoders are neural networks that learn compact representations of complex data and are widely used for dimensionality reduction, feature extraction, anomaly detection, and generative modelling. In Earth system science, they can be applied to large climate and weather datasets to identify patterns, compress high-dimensional data, and support a range of downstream machine learning tasks. The session is aimed at science students, early-career scientists, or later-career scientists who want a quick orientation into the world of machine learning. Data scientists may also find it interesting from the perspective of the application to Earth system science. The first session will be a presentation covering neural network architectures and machine learning training processes, to explain the terms and concepts involved. The PyEarthTools framework will be introduced, and some attendees can make an early start on using the worked examples. The second session will give attendees the chance to train their own models, either by working through the examples included in a series of Jupyter notebooks, or by connecting their own data sources into pre-defined network architectures in similar fashion. A basic knowledge of the NCI ARE environment is recommended. Attendees can attend both sessions, or choose the one most of interest. ---
**Author:** ![Tennessee](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/tennessee/32/2245_2.png) [@Tennessee](https://forum.access-hive.org.au/u/Tennessee)\ **Post date:** [16 August 2026 23:39 UTC](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2 "2026-08-16T23:39:25Z")
> **ML Workshop 2026 Technical Setup Instructions for Autoencoder Training Sessions** ### Introduction: This post covers the technical setup requirements for the hands-on training session on: **Autoencoders: Part 2** | 2026-08-20T03:30:00Z→2026-08-20T05:00:00Z. We will go through the rest of these setup instructions together at the end of the Autoencoders: Part 1 training session, so it’s not essential to complete these steps 3-4 in advance. However, if you’d like to maximise your hands-on time during the workshop, working through these steps beforehand will give you a head start. * * * ### Table of Contents: > ‼ **The first two steps must be completed prior to the training sessions**. 1. [**Sign up for an NCI account**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-1-sign-up-for-nci-account-4) 2. [**Join up to the relevant NCI projects**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-2-join-nci-projects-5) * * * > 📝 **Setup instructions: We’ll go together at the end of Day 1 Training** 1. [**Create your working directories and copy the training notebooks**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-3-create-your-working-directories-and-copy-the-training-notebooks-7) 2. [**Set up a JupyterLab session in NCI ARE**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-4-set-up-jupyterlab-in-nci-are-8) 3. [**Test your setup**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-5-test-your-setup-9) > 🚀 **On the day: Start the session 30 mins prior** 1. [**Launch JupyterLab early on the day**](https://forum.access-hive.org.au/t/tennessee-leeuwenburg-auto-encoder-training-sessions/6788/2#p-23841-h-6-launch-jupyterlab-early-on-the-day-10) Preferably 30 minutes prior * * * ### Required setup before the training session * * * #### 1. Sign up for NCI account If you don’t already have an NCI account, [sign up here](https://my.nci.org.au/mancini/signup/0). When creating your account, you can nominate project `nf33` as your main project. > **New to NCI ARE?** > The practical exercises will use the [NCI Australian Research Environment (ARE)](https://docs.access-hive.org.au/getting_started/are/). Basic familiarity with the ARE environment is recommended, but we’ll also step through the setup together during the training. * * * #### 2. Join NCI projects Please request membership of the following NCI projects. - [dk92](https://my.nci.org.au/mancini/project/dk92/join): Data science environment, used to access the PyEarthTools Python modules. - [nf33](https://my.nci.org.au/mancini/project/nf33/join): ACCESS-NRI training environment, used for compute and setting up your working directory. - [rv74](https://my.nci.org.au/mancini/project/rv74/join): Himawari 8 data used in the Auto-encoder examples. - [rt52](https://my.nci.org.au/mancini/project/rt52/join): Required to fully participate in the workshop. * * * - Optional [wb00](https://my.nci.org.au/mancini/project/wb00/join): For accessing ERA5 on-disk data without needing to download it. - Optional [rq0](https://my.nci.org.au/mancini/project/rq0/join): For radar-based precipitation data. > **Note** : `rt52` project was not listed in the original setup email, but is required to fully participate in the workshop. Please make sure you request access to this project as well. * * * ### Setup instructions: We’ll go through at the end of the Part 1 training * * * #### 3. Create your working directories and copy the training notebooks Once your NCI account and project access are ready, log in to Gadi from a terminal: ```auto ssh gadi ``` Move to the `nf33` project directory: ```auto cd /g/data/nf33 ``` Create a directory using **your own NCI username** , then move into it: ```auto mkdir -p cd ``` For example, if your NCI username were `ab1234`: ```auto mkdir -p ab1234 cd ab1234 ``` You can check your current location with: ```auto pwd ``` It should look similar to: ```auto /g/data/nf33/ ``` Now create the directories you’ll need for the workshop: ```auto mkdir -p userbase62 notebooks ``` Your directory should now contain: ```auto userbase62/ notebooks/ ``` Now, we are done with the creation of working directories! Let’s copy over the **training notebooks**. **Copy the training notebooks:** Move into your notebooks directory: ```auto cd notebooks ``` Copy the workshop notebooks from the shared training directory: ```auto cp -r /g/data/nf33/autoencoder_notebooks/* . ``` Check that the files have copied successfully: ```auto ls -l ``` You should see files including the workshop notebooks and a `tutorial` directory. > **Example Output for `ls -l`:** > > ```auto > [ab1234@gadi-login-08 notebooks\]$ ls -l > > total 696 > > rw-r----- 1 ab1234 nf33 523602 Aug 12 17:04 DataAccessForWorkshopsNCI.ipynb > rw-r----- 1 ab1234 nf33 184259 Aug 12 17:04 TemporalWindowDemo.ipynb > drwxr-sr-x 2 ab1234 nf33 4096 Aug 12 17.04 tutorial > > ``` * * * #### 4. Set up JupyterLab in NCI ARE Open [NCI ARE](https://are.nci.org.au/) and select **JupyterLab**. **Main JupyterLab settings:** Recommended fields are as follows: | Setting | Value | | --- | --- | | **Walltime** | `3` hours | | **Queue** | `gpuvolta` | | **Compute Size** | `1gpu` | | **Project** | `nf33` | | **Storage** | See below | For **Storage** , add the project directories required by the notebooks: ```auto gdata/dk92+gdata/nf33+gdata/wb00+gdata/rv74+gdata/rq0+gdata/rt52 ``` > **Main JupyterLab settings** > > ![Screenshot 2026-08-18 at 09.07.27](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/e/ee248d2a9c95005fd32f0b1e45d39e1320d357c3.png) **Advanced Settings:** Tick **Show advanced settings**. For **Module directories** , enter: ```auto /g/data/dk92/apps/Modules/modulefiles ``` For **Modules** , enter: ```auto pet/0.6.2 ``` > **Advanced Settings:** > > ![image004](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/7/731f5a9367de68e2f7a411718da3810ffdfffaed.png) Further down the advanced settings, find **Environment variables**. Enter: ```auto PYTHONUSERBASE="/g/data/nf33//userbase62/" ``` Replace `` with your actual NCI username. For example: ```auto PYTHONUSERBASE="/g/data/nf33/abc123/userbase62/" ``` Set **Jobfs size** to: ```auto 100GB ``` > **Advanced Settings \> Environment Variables:** > > ![image005](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/f/f2d237ccf978d310f63d238fef014827d3b09a32.png) > **Tip:** Once you’ve confirmed everything is working, you can use **Save settings** in ARE so you don’t need to enter all of the configuration again on the day. 🚀 When you’re ready, select **Launch**. It may take a few minutes for your JupyterLab session to start. * * * #### 5. Test your setup Once the JupyterLab session has started, click Open JupyterLab button under My Interactive Sessions tab. > Click Open JupyterLab > > ![Screenshot 2026-08-18 at 09.40.55](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/3/35b40318dea2fddacdbd5ae7c8745ce8cbe53217.png) Use the file browser to navigate to: ```auto /g/data/nf33//notebooks ``` Open and run the cells for: ```auto DataAccessForWorkshopsNCI.ipynb ``` This gives us a chance to check that the environment and data access are all working before moving on to the main exercises. * * * #### 6. Launch JupyterLab early on the day Before we move into the hands-on component, we’ll make sure everyone’s JupyterLab session is running. ARE jobs can sometimes spend a little time waiting in the queue, so we’ll aim to launch the sessions around **30 minutes** before we need them. Once your session starts, navigate to: ```auto /g/data/nf33//notebooks/tutorial ``` and open: ```auto IntroductionForWorkshops.ipynb ``` From there, we’re ready to start the exercises. ✅