Installing ACCESS-OM2 on NeSI (New Zealand supercomputer)

COSIMA
, , ,
1

Hi COSIMA team,
I am an investigator on a recently awarded Marsden Fund grant, led by Shaun Eaves at Victoria University of Wellington. In the grant we propose to use ACCESS-OM2-025 as part of our experiment plan.
We are just trying to get set up on NeSI, the NZ supercomputer, and looking to install the various components.
I am basically the person responsible for setting this up, since I have prior experience with installing some parts of the model on different clusters. But, before I dive in and start trying to do it myself, I wanted to check in and see if ACCESS-NRI has any capacity / interest to provide advice on this?

I saw this previous thread on installing ACCESS-OM2 on Leonardo. Would be super appreciated if I could touch base with folks at ACCESS-NRI on how to ensure a proper set up on NeSI.

Also, NeSI uses SLURM rather than PBS… again I can dive in and try to configure this myself (I did something similar with a much older version of payu), but any capacity to pick your brains would be helpful.

Regards,
Dave

2

Hi Dave,

Awesome to hear about the grant and your interest in getting OM2 over to NeSI!

before I dive in and start trying to do it myself, I wanted to check in and see if ACCESS-NRI has any capacity / interest to provide advice on this?

AFAIK, this is not something we officially support at the present time and the release team are low on people-time at the moment. Still, as you’ve seen with Leonardo progress was possible with some perseverance as it has also been on Setonix (Pawsey).

Also, NeSI uses SLURM rather than PBS… again I can dive in and try to configure this myself (I did something similar with a much older version of payu), but any capacity to pick your brains would be helpful.

A little work has been done on this in the past. Here’s a comment from @jo-basevi:

This is Aidans PR from a couple years ago to get something working on pawsey- Porting to pawsey by aidanheerdegen · Pull Request #326 · payu-org/payu · GitHub - related to this issue. I think that is more likely likely to work than my draft PR which is only just using the slurm client from hpcpy so far.

@ChrisC28 and @john_reilly also have a hacked version of payu from a while back that has limited functionality (I’m told) on Pawsey, given that system also uses Slurm that might help. HPCpy is probably the better route to go long-term as it’s intended to be scheduler agnostic.

As it happens, myself @helen @harshula have been working on similar things (OM3) over on Setonix recently. So below are a few things that might help you get started/have a clearer sense of what’s involved. I’d expect the general workflow to be similar-ish for OM2/OM3.

I’ve got a mega-issue here that collates the various issues. My latest problem is likely related to not setting up the run folder correctly but before that I had some build issues. You can see my very scrappy notes here (these arn’t meant to be used by anyone else just yet!).

Perhaps the most useful thing to look at to start is the build changes that were made for Setonix. In the Setonix case, we’re lucky that spack is already installed on the system. You might have to install it, if it’s not there already.

Chris

2 Likes
3

Thanks a lot @cbull. That is a good starting point and I will see how far I get based on these notes.

4

Hi @dkhutch ,

Does NeSI have Spack installed? Does it have Intel or AMD CPUs? Which MPI implementation is installed as the system MPI? Which compilers are available and which compiler is preferred?

5

Hi @harshula,

  1. Yes Spack is available as a module on NeSI.
  2. It uses AMD processors.
  3. So far I’ve got a test compile script for MOM5.1.0 (not all of ACCESS-OM2) to “work” with Intel/2022.0.2 compilers with Intel MPI wrappers. However I’ve not tested running it! That is the one I’ll try to use because NeSI comes with pre-installed NetCDF-fortran library module with the same compilers.
6

Hi @dkhutch ,

Can you please module load spack, then run the following commands and provide us the output of:

  1. spack -V
  2. spack compiler list
  3. spack providers mpi
7

So far spack -V causes the terminal to freeze for 10+ min. Not looking promising!!
Perhaps I might have to install my own version.

1 Like
8

Hi @dkhutch ,

Oh no! Does module avail spack display the versions that are installed?

9

Ok so given that the module spack seemed broken, I installed my own version of spack using this.

Then with the Intel MPI modules loaded, I did:
spack compiler find
Which seemed to find some things… Followed this with
spack compiler list
That gave me this output:

==> Available compilers
-- gcc rocky9-x86_64 --------------------------------------------
[e]  gcc@11.4.1  [e]  gcc@11.3.0

-- intel-oneapi-compilers-classic rocky9-x86_64 -----------------
[e]  intel-oneapi-compilers-classic@2021.5.0

THen I did this:
spack providers mpi
And got this output:

mpi:
-- no arch / no compilers ---------------------------------------
cray-mpich        mpi-serial  mpich@:3.2     mpt@1:        mvapich2       openmpi@:1.2          spectrum-mpi
cray-mvapich2     mpich@:1.0  mpich          mpt@3:        mvapich2@2.1:  openmpi@1.3:1.7.2
fujitsu-mpi       mpich@:1.1  mpilander      msmpi         mvapich2@2.3:  openmpi@1.7.3:1.7.4
hpcx-mpi          mpich@:1.2  mpitrampoline  mvapich       nmad           openmpi@1.7.5:1.10.7
intel-oneapi-mpi  mpich@:3.1  mpt            mvapich-plus  nvhpc          openmpi@2.0.0:

But as mentioned, NeSI modules do provide an Intel MPI compiler. If I load the compiler module that I’m intending to use, i.e.
module load impi/2021.5.1-intel-compilers-2022.0.2

It loads a whole suite of modules, i.e. I have this:

Currently Loaded Modules:
  1) NeSI/zen3                               (S)   9) impi/2021.5.1-intel-compilers-2022.0.2
  2) GCCcore/11.3.0                               10) iimpi/2022a
  3) zlib/1.2.11-GCCcore-11.3.0                   11) Szip/2.1.1-GCCcore-11.3.0
  4) binutils/2.38-GCCcore-11.3.0                 12) HDF5/1.12.2-iimpi-2022a
  5) intel-compilers/2022.0.2                     13) LegacySystemLibs/.crypto7              (H)
  6) libpmi/2-slurm                               14) cURL/7.83.1-GCCcore-11.3.0
  7) numactl/2.0.14-intel-compilers-2022.0.2      15) netCDF/4.8.1-iimpi-2022a
  8) UCX/1.12.1-intel-compilers-2022.0.2          16) netCDF-Fortran/4.6.0-iimpi-2022a

Most of those came automatically packaged together. I did load the netCDF-Fortran one, which auto-loaded the HDF5 and netCDF/4.8.1-iimpi-2022a one.

10

Returns the following output below. I tried both of them and they seemed to just freeze out.

------------------------------------------ /opt/nesi/lmod/mahuika -------------------------------------------
   Spack/0.17.0    Spack/0.17.2 (D)
11

HI @dkhutch ,

Spack v0.17.x is really old. Probably better not to use it. On Gadi we use Spack v0.22.5 and Setonix uses v0.23.1. We are planning to migrate to Spack v1.x.

Is there a way I could get login access?

12

I will send a request. Hopefully will hear back tomorrow.

1 Like
13

@dkhutch just noticed some of my links in that post were out of order, now fixed I hope.

1 Like
14

Hi Dave!

Please have a look at my repository (branch _leonardo):

It contains the ACCESS-OM2 1° configuration adapted for Leonardo (SLURM) with detailed instructions on how to install and run the model. It also has a link to my repo with payu adaptation for SLURM.
I’ve also added optional profiling hooks for payu (mPIP and perf), which can be switched on and off.

The repo includes my master’s thesis as well, with detailed explanations of the porting and profiling of ACCESS-OM2.

To run the 0.25° configuration, you only need to download the initial conditions for the 025 experiment locally.

I’d be very happy to share my experience - it was quite a journey last year, with great help from the ACCESS-NRI and COSIMA communities.
If there’s anything I can help with, please post it here or DM me.

@dkhutch

Natalia

4 Likes
15

Thanks very much Natalia! I will check out the changes you have made and see if this can help us get up and running.

1 Like
16

Hi all,
Just to note here that I tried going through the steps documented in Natalia’s Github repo. (Thanks for the nice clear explanation @Natalia.)
I was following the instructions to use spack/0.23
Unfortunately it choked when I tried to do any kind of spack install, e.g. this one:

spack install intel-oneapi-compilers@2021.2.0 target=x86_64

The error message was:

Error: Version requirement 3.31.6 on cmake for cmake cannot match any known version from package.py or externals

And if I tried looking for cmake using spack info cmake, it would list a bunch of versions with the latest one being 3.30.5. So, I’m guessing this means I’d have to use a newer version of spack? But then I don’t know if that would ruin the installation processes…

I believe @harshula might be updating the access-om2 to use spack/1.1, in which case should I just wait until that is available?

Regards,
David

1 Like
17

Yes, I think you are correct, we are updating at the moment.

I was told recently to stick with the older version for the time being but I think you’re right @harshula is the person to chat to.

18

Yep… if I use a later version of spack such as 1.1, then it can find the newer cmake but it can’t find access-om2. As I’m very much unfamiliar with spack, I will pause for now and look out for guidance once the migration to 1.1 is done.

19

Hi Dave,

At this stage you are dealing with spack internal packages (https://packages.spack.io) from the spack builtin catalog, not ACCESS-NRI catalog yet.

I guess the error that you see could be (?) due to the deprecation of 2021.2.0 intel-oneapi, idk.

I just tried this and it worked out on Leonardo, maybe you can try to reproduce your installation with spack@1.0 and intelapi@2025.0.4?

git clone -c feature.manyFiles=true https://github.com/spack/spack.git --branch releases/v1.0 --single-branch --depth=1
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch main
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main


ln -s -r -v spack-config/v1.0/ci/* spack/etc/spack/
. spack-config/spack-enable.bash 
spack env create accessom2_test

creation of env takes some time

After that check catalogs:

[ntilinin@login07 ACCESS-NRI_test]$ spack repo list
[+] access.nri    v2.0    /leonardo/home/userexternal/ntilinin/ACCESS-NRI_test/package_repos/3t2klr5/spack_repo/access/nri
[+] builtin       v2.2    /leonardo/home/userexternal/ntilinin/ACCESS-NRI_test/package_repos/fncqgg4/repos/spack_repo/builtin

builtin is the one where spack gets the intel-oneapi-compilers from. I tried to install the later version of intelapi:

spack install intel-oneapi-compilers@2025.0.4 target=x86_64

it worked, and them we move to the ACCESS-NRI catalog:

spack install access-om2 ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.1.4 %intel-oneapi-compilers@2025.0.4 target=x86_64

Takes around 40 min.

It worked in my case. cmake 3.31.6 is the dependency of access-om2 that spack installs by itself:

==> No binary for cmake-3.31.6-kjnvzbux3wezwjl3teapgkytyly6k6fu found: installing from source
==> Installing cmake-3.31.6-kjnvzbux3wezwjl3teapgkytyly6k6fu [37/59]
==> Fetching https://mirror.spack.io/_source-cache/archive/65/653427f0f5014750aafff22727fb2aa60c6c732ca91808cfb78ce22ddd9e55f0.tar.gz
    [100%]   11.71 MB @    9.3 MB/s
==> Fetching https://mirror.spack.io/_source-cache/archive/fd/fdea723be9713f3ed4624055bf21ef5876647d63c151b91006608ec44a912ae1
    [100%]    1.19 KB @    3.4 MB/s
==> Applied patch https://github.com/kitware/cmake/commit/1b0c92a3a1b782ff3e1c4499b6ab8db614d45bcd.patch?full_index=1
==> cmake: Executing phase: 'bootstrap'
==> cmake: Executing phase: 'build'
==> cmake: Executing phase: 'install'
==> cmake: Successfully installed cmake-3.31.6-kjnvzbux3wezwjl3teapgkytyly6k6fu
  Stage: 3.19s.  Bootstrap: 1m 14.80s.  Build: 2m 8.66s.  Install: 12.62s.  Post-install: 4.70s.  Total: 3m 44.16s
[+] /leonardo/home/userexternal/ntilinin/ACCESS-NRI_test/release/linux-x86_64_v4/cmake-3.31.6-kjnvzbux3wezwjl3teapgkytyly6k6fu

In the end the whole installation is successful:

==> Installing access-om2-latest-4occllpmflt225nbc3qe26bg7ajt2aow [59/59]
==> No patches needed for access-om2
==> access-om2: Executing phase: 'install'
==> access-om2: Successfully installed access-om2-latest-4occllpmflt225nbc3qe26bg7ajt2aow
  Stage: 0.00s.  Install: 0.00s.  Post-install: 0.07s.  Total: 0.13s

Load the whole thing with spack and then executables seen by the system:

[ntilinin@login07 ACCESS-NRI_test]$ spack load access-om2@latest 
[ntilinin@login07 ACCESS-NRI_test]$ cice_auscom_
cice_auscom_1440x1080_48x40_480p.exe     cice_auscom_3600x2700_200x180_1682p.exe  cice_auscom_3600x2700_90x90_722p.exe     cice_auscom_360x300_24x1_24p.exe

Hope this helps!

Natalia

1 Like
20

@harshula informed me that the set of versions now is:

spack 1.1

intel@2021.10.0

Will try to reinstall everything later on.

1 Like