ACCESS-NRI Analysis3 Conda Environments – New Release Announcement

ACCESS-NRI Releases
, , , ,
1

:rocket: :rocket: Announcing the New ACCESS-NRI conda/analysis3 Environments

We are pleased to announce that the ACCESS-NRI conda environments, including the analysis3 environment formerly maintained by CLEX on project hh5, are now hosted on project xp65 on NCI Gadi. Users must request access to project xp65 via their NCI dashboard. The environments are accessible via modules and can be loaded as follows:

module use /g/data/xp65/public/modules
module load conda/analysis3

:warning: Key Changes and Improvements

:white_check_mark: Environments are now managed using a containerised file system for better stability and consistency.
:white_check_mark: Regular updates will be provided for the active monthly environments.
:white_check_mark: Clear distinction between stable and cutting-edge environments.
:white_check_mark: All dependencies on the former hh5 project and COECMS conda channel have been removed. We recommend users adjust their workflows accordingly.

:fire: Two Series of Environments: Analysis3 and Analysis3-Edge

We are releasing two parallel series of environments:

  1. analysis3-YY.MM – A stable environment designed for long-term usability with well-tested, reliable package versions.
  2. analysis3-edge-YY.MM – A cutting-edge environment with the latest available packages, ideal for those needing the most up-to-date software and features.

These series offer flexibility: use analysis3 for stability, or analysis3-edge for the latest updates.

:statue_of_liberty: Environment Updates and Freezing Policy

To ensure stability, updates will follow these guidelines:

  • The current month’s environments (analysis3-YY.MM and analysis3-edge-YY.MM) can be updated on demand throughout the month (e.g., analysis3-25.04 and analysis3-edge-25.04 in April).
  • At the end of the month, environments will be frozen and no longer updated.
  • Frozen environments will remain available for at least 6 months.
  • A new set of environments will be created each month with the latest tested packages.

Once frozen, users can rely on these versions without unexpected changes, while those needing newer versions can transition to the latest available monthly release.

:light_bulb: How to update workflows to use the new conda/analysis3 environment

The only usage difference is to replace instances of hh5 with xp65. More detailed instructions can be found on the conda/analysis3 page of the ACCESS-Hive Docs.

Examples of loading the new conda/analysis3 environment include:

  • On the command line, for each new session, run:
module use /g/data/xp65/public/modules
module load conda/analysis3
  • If you need to use the conda/analysis3 environment for a PBS job (including for usage with rose/cylc), in addition to adding the two module use and module load lines above, add the xp65 project (or replace hh5 with xp65):
#PBS -l storage=gdata/xp65
  • When using Australian Research Environment (ARE), add gdata/xp65 to “Storage”, /g/data/xp65/public/modules/ to “Module directories” (under “Advanced options”), and add conda/analysis3 to “Modules”. Additionally, users can switch kernels inside their ARE JupyterLab instance to select the appropriate environment.

Note that by not specifying a version (e.g. using conda/analysis3 rather than conda/analysis3-YY.MM), the latest frozen environment will be loaded. We encourage those who want the newest features to help us test the current version by explicitly loading this month’s environment.

If you previously added the hh5 conda environment to your .bashrc or .bash_profile files, we recommend that you remove those lines rather than replacing with xp65, as this can have some unforeseen interference with other processes you might run on Gadi. We suggest instead that you explicitly load the environment for each new session (the first bullet point above).

:red_question_mark: Requesting Additional Packages

To request additional packages, users should open an issue on the ACCESS-Analysis-Conda GitHub repository.

Package requests will be assessed based on:

  • The dependencies required and their impact on the overall package stack.
  • Whether the package is actively maintained on GitHub.
  • The value of the package to the broader community.

Packages with no additional dependencies will generally be added, while those that rely on outdated SPEC0 packages will not be included due to compatibility issues.

New environments incorporate the latest releases of SPEC0-listed packages. The analysis3-YY.MM environment will always include the most recent versions where possible.

Python is an exception: the goal is to include the latest supported version (currently Python 3.11, within the 3.10-3.13 range). Minor version updates are safe, but major changes may introduce incompatibilities.

Maintainers will balance new features with package compatibility. For example, NumPy 2.0 is a significant update, so:

  • analysis3-YY.MM will retain NumPy 1.x until key packages transition.
  • analysis3-edge-YY.MM will feature NumPy 2.x for those needing the latest updates.

Packages that introduce minimal dependencies will be added. However, if a package requires significant changes or downgrades a “key” package, a formal request justifying its importance to the community must be submitted.

Key SPEC0 Packages:

  • Python
  • Dask
  • NumPy
  • SciPy
  • Matplotlib
  • Pandas
  • Scikit-Image
  • Scikit-Learn
  • Xarray
  • IPython
  • Zarr

:sos_button: Support and Further Information

For support or for general feedback, users can create a topic in Technical > Infrastructure and tag it with help and conda. (For more details on getting help, see “How do you get help and support?)”.

For more detailed usage instructions, see the conda/analysis3 page on the ACCESS-Hive Docs.

We encourage users to explore the new environments and provide feedback to ensure they meet community needs.

We would like to acknowledge CLEX support and funding, and the dedicated work of the CMS team for establishing and maintaining this Python environment for the community. It is by leveraging their efforts that the ACCESS-NRI team is able to continue to support this important community resource.

3 Likes