How to build ACCESS-OM2 on Gadi

Installation instructions for model developers on Gadi v6

[22/02/2024: depend on openmpi and not nci-openmpi]
Known Issues: (Setup-env scripts bug on back and forth between different spack instances/clones · Issue #40901 · spack/spack · GitHub)

  1. Login to Gadi:
ssh USERNAME@gadi.nci.org.au
  1. Clone the relevant git repositories (read-only):
git clone -c feature.manyFiles=true https://github.com/spack/spack.git --branch releases/v0.20 --single-branch --depth=1
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch main
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main
  1. Link spack-config configurations files to the Spack instance:
ln -s -r -v spack-config/v0.20/gadi/* spack/etc/spack/
  1. Set Spack environment variables (required after each login):
. spack-config/spack-enable.bash
  1. Build ACCESS-OM2 (initialisation may take a long time without any output):
spack install access-om2 ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.0.2 %intel@19.0.5.281

Instructions for general users on Gadi v3

[30/01/2024: renamed spack-start.bash to spack-enable.bash in deployment]

  1. Request membership of vk83.

  2. Login to Gadi:

ssh USERNAME@gadi.nci.org.au
  1. Set Spack environment variables:
. /g/data/vk83/apps/spack/0.20/spack-config/spack-enable.bash
  1. Display pre-built packages:
spack find
1 Like

If the spack install ... command is failing with the following error:

==> Installing cmake-3.26.3-4hg4d2a6d72efysvvyfkepnvl6khes3o
==> No binary for cmake-3.26.3-4hg4d2a6d72efysvvyfkepnvl6khes3o found: installing from source
==> Fetching file:///g/data/<PROJECT>/<USERNAME>/spack-mirror-access-om2/_source-cache/archive/bb/bbd8d39217509d163cb544a40d6428ac666ddc83e22905d3e52c925781f0f659.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
Killed

you may be using v1 of the instructions where CMake was being built by Spack. To avoid this problem, please use v2 of the instructions where the system CMake is used

@harshula what is meant by ... here? Where do I find what should go there?

  1. What’s an Ellipsis? Definition and Examples | Grammarly Blog
  2. The previous step would have auto-populated the remainder of that section.

I have just used the instructions to successfully build on Gadi login at /g/data/tm70/pcl851/spack

I couldn’t get it to work - build logs here /scratch/v45/aek156/tmp/spack-stage/

Hi @aekiss, I don’t have permission to access v45. Let’s go through the steps together when you’re in the office.

Hi @aekiss ,

Your Spack instance was unable to compile the required build tools. e.g. gmake:

==> gmake: Executing phase: 'build'
==> [2023-11-06-21:27:41.679695] '/scratch/[...]/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/build.sh'
config.status: creating ./lib/alloca.h__
compiling lib/concat-filename.c...
In file included from /usr/include/bits/floatn.h(119),
                 from /usr/include/stdlib.h(55),
                 from /scratch/v45/aek156/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/lib/concat-filename.h(20),
                 from /scratch/v45/aek156/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/lib/concat-filename.c(22):
/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;

Looks like a problem with your compiler config.

Do you remember why you made the following changes to your .spack/linux/compilers.yaml?

@@ -13,9 +13,22 @@
     environment: {}
     extra_rpaths: []
 - compiler:
+    spec: gcc@=13.2.0
+    paths:
+      cc: /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/gcc
+      cxx: null
+      f77: null
+      fc: null
+    flags: {}
+    operating_system: rocky8
+    target: x86_64
+    modules: []
+    environment: {}
+    extra_rpaths: []
+- compiler:
     spec: gcc@=8.5.0
     paths:
-      cc: /opt/nci/bin/gcc
+      cc: /bin/gcc
       cxx: /opt/nci/bin/g++
       f77: /opt/nci/bin/gfortran
       fc: /opt/nci/bin/gfortran

Perhaps move your .spack to another location and try the instructions with a new empty .spack.

spack find picked up gcc@13.2.0 because I had the conda/analysis3-23.07 module loaded.

It builds successfully if I do module purge before step 1.

Thanks @aekiss for testing and providing feedback. As a result, I’m planning on disabling local config (i.e. ~/.spack) to avoid similar issues.

5 posts were split to a new topic: Modifying ACCESS-OM2 spack package

Slightly peripheral question, but the spack.yaml that defines the model uses git tags to identify the model component:

spack:
  # add package specs to the `specs` list
  specs:
  - access-om2@git.2023.11.23
  packages:
    cice5:
      require: '@git.2023.10.19'
    mom5:
      require: '@git.2023.11.09'
    libaccessom2:
      require: '@git.2023.10.26'
    oasis3-mct:
      require: '@git.2023.11.09'
    netcdf-c:
      require: '@4.7.4'
    netcdf-fortran:
      require: '@4.5.2'
    parallelio:
      require: '@2.5.2'
    nci-openmpi:
      require: '@4.0.2'
    all:
      compiler: [intel@19.0.5.281]

That means having installed all the access-om2 components with spack the versions of the model components are not displayed in the same way as components installed by version

$ spack find -vl
-- linux-rocky8-x86_64 / intel@19.0.5.281 -----------------------
...
nyxvikk json-fortran@8.3.0~ipo build_system=cmake build_type=Release generator=make
ajlvl3k libaccessom2@master~deterministic~ipo~optimisation_report build_system=cmake build_type=Release generator=make
uznvr36 mom5@master~deterministic~optimisation_report build_system=makefile
i3inxza netcdf-c@4.7.4~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools

e.g. json-fortran@8.3.0 vs mom5@master

I couldn’t find a way get the git tag value, for example for mom5 it would be 2023.11.09.

Please use a new topic. Did you use the spack.yaml to install the packages? This is how it should look:

$ spack find
-- linux-rocky8-x86_64 / intel@19.0.5.281 -----------------------
access-om2@git.2023.11.23=2023.11.23    mom5@git.2023.11.09=2023.11.09
cice5@git.2023.10.19=2023.10.19         nci-openmpi@4.0.2
cmake@3.24.2                            netcdf-c@4.7.4
datetime-fortran@1.7.0                  netcdf-fortran@4.5.2
gmake@4.4.1                             oasis3-mct@git.2023.11.09=2023.11.09
hdf5@1.14.1-2                           parallelio@2.5.10
json-fortran@8.3.0                      pkgconf@1.9.5
libaccessom2@git.2023.10.26=2023.10.26  zlib@1.2.13
==> 16 installed packages