How to build ACCESS-OM2 on Gadi

harshula · 25 October 2023 03:11

Known Issues with Spack v0.20: (Setup-env scripts bug on back and forth between different spack instances/clones · Issue #40901 · spack/spack · GitHub)

Instructions for general users on Gadi v3

[30/01/2024: renamed spack-start.bash to spack-enable.bash in deployment]

Request membership of vk83.
Login to Gadi:

ssh USERNAME@gadi.nci.org.au

Set Spack environment variables:

. /g/data/vk83/apps/spack/0.20/spack-config/spack-enable.bash

Display pre-built packages:

spack find

Installation instructions for model developers on Gadi v7

[23/04/2024: Use the ACCESS-NRI fork of spack]

Login to Gadi:

ssh USERNAME@gadi.nci.org.au

Clone the relevant git repositories (read-only):

git clone -c feature.manyFiles=true https://github.com/ACCESS-NRI/spack.git --branch releases/v0.20 --single-branch --depth=1
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch main
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main

Link spack-config configurations files to the Spack instance:

ln -s -r -v spack-config/v0.20/gadi/* spack/etc/spack/

Set Spack environment variables (required after each login):

. spack-config/spack-enable.bash

Build ACCESS-OM2 (initialisation may take a long time without any output):

spack install access-om2 ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.0.2 %intel@19.0.5.281

dev-build instructions for model developers on Gadi v6

[23/04/2024: Use the ACCESS-NRI fork of spack]

Login to Gadi:

ssh USERNAME@gadi.nci.org.au

Clone the relevant git repositories (read-only):

git clone -c feature.manyFiles=true https://github.com/ACCESS-NRI/spack.git --branch releases/v0.20 --single-branch --depth=1
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch main
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main

Link spack-config configurations files to the Spack instance:

ln -s -r -v spack-config/v0.20/gadi/* spack/etc/spack/

Set Spack environment variables (required after each login):

. spack-config/spack-enable.bash

For example, do a spack dev-build inside the oasis3-mct git directory (initialisation may take a long time without any output):

spack install --only dependencies oasis3-mct@git.e9640e767ac9f1fad71ddaafa7bbe3f12e8091b6 ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^openmpi@4.0.2 %intel@19.0.5.281
git clone https://github.com/ACCESS-NRI/oasis3-mct.git
cd oasis3-mct
spack dev-build oasis3-mct@git.e9640e767ac9f1fad71ddaafa7bbe3f12e8091b6 ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.0.2 %intel@19.0.5.281

harshula · 26 October 2023 10:33

If the spack install ... command is failing with the following error:

==> Installing cmake-3.26.3-4hg4d2a6d72efysvvyfkepnvl6khes3o
==> No binary for cmake-3.26.3-4hg4d2a6d72efysvvyfkepnvl6khes3o found: installing from source
==> Fetching file:///g/data/<PROJECT>/<USERNAME>/spack-mirror-access-om2/_source-cache/archive/bb/bbd8d39217509d163cb544a40d6428ac666ddc83e22905d3e52c925781f0f659.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
Killed

you may be using v1 of the instructions where CMake was being built by Spack. To avoid this problem, please use v2 of the instructions where the system CMake is used

aekiss · 2 November 2023 03:09

@harshula what is meant by ... here? Where do I find what should go there?

harshula · 2 November 2023 04:38

What’s an Ellipsis? Definition and Examples | Grammarly Blog
The previous step would have auto-populated the remainder of that section.

paulleopardi · 2 November 2023 05:45

I have just used the instructions to successfully build on Gadi login at /g/data/tm70/pcl851/spack

aekiss · 6 November 2023 21:43

I couldn’t get it to work - build logs here /scratch/v45/aek156/tmp/spack-stage/

harshula · 6 November 2023 23:14

Hi @aekiss, I don’t have permission to access v45. Let’s go through the steps together when you’re in the office.

harshula · 12 November 2023 22:36

Hi @aekiss ,

Your Spack instance was unable to compile the required build tools. e.g. gmake:

==> gmake: Executing phase: 'build'
==> [2023-11-06-21:27:41.679695] '/scratch/[...]/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/build.sh'
config.status: creating ./lib/alloca.h__
compiling lib/concat-filename.c...
In file included from /usr/include/bits/floatn.h(119),
                 from /usr/include/stdlib.h(55),
                 from /scratch/v45/aek156/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/lib/concat-filename.h(20),
                 from /scratch/v45/aek156/tmp/spack-stage/spack-stage-gmake-4.4.1-h5uwuoe6h3nknjnagjbskugtwch7q5c3/spack-src/lib/concat-filename.c(22):
/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;

Looks like a problem with your compiler config.

Do you remember why you made the following changes to your .spack/linux/compilers.yaml?

@@ -13,9 +13,22 @@
     environment: {}
     extra_rpaths: []
 - compiler:
+    spec: gcc@=13.2.0
+    paths:
+      cc: /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/gcc
+      cxx: null
+      f77: null
+      fc: null
+    flags: {}
+    operating_system: rocky8
+    target: x86_64
+    modules: []
+    environment: {}
+    extra_rpaths: []
+- compiler:
     spec: gcc@=8.5.0
     paths:
-      cc: /opt/nci/bin/gcc
+      cc: /bin/gcc
       cxx: /opt/nci/bin/g++
       f77: /opt/nci/bin/gfortran
       fc: /opt/nci/bin/gfortran

Perhaps move your .spack to another location and try the instructions with a new empty .spack.

aekiss · 14 November 2023 00:16

spack find picked up gcc@13.2.0 because I had the conda/analysis3-23.07 module loaded.

It builds successfully if I do module purge before step 1.

harshula · 14 November 2023 01:13

Thanks @aekiss for testing and providing feedback. As a result, I’m planning on disabling local config (i.e. ~/.spack) to avoid similar issues.

Aidan · 22 February 2024 06:54

5 posts were split to a new topic: Modifying ACCESS-OM2 spack package

Aidan · 23 February 2024 05:27

Slightly peripheral question, but the spack.yaml that defines the model uses git tags to identify the model component:

spack:
  # add package specs to the `specs` list
  specs:
  - access-om2@git.2023.11.23
  packages:
    cice5:
      require: '@git.2023.10.19'
    mom5:
      require: '@git.2023.11.09'
    libaccessom2:
      require: '@git.2023.10.26'
    oasis3-mct:
      require: '@git.2023.11.09'
    netcdf-c:
      require: '@4.7.4'
    netcdf-fortran:
      require: '@4.5.2'
    parallelio:
      require: '@2.5.2'
    nci-openmpi:
      require: '@4.0.2'
    all:
      compiler: [intel@19.0.5.281]

That means having installed all the access-om2 components with spack the versions of the model components are not displayed in the same way as components installed by version

$ spack find -vl
-- linux-rocky8-x86_64 / intel@19.0.5.281 -----------------------
...
nyxvikk json-fortran@8.3.0~ipo build_system=cmake build_type=Release generator=make
ajlvl3k libaccessom2@master~deterministic~ipo~optimisation_report build_system=cmake build_type=Release generator=make
uznvr36 mom5@master~deterministic~optimisation_report build_system=makefile
i3inxza netcdf-c@4.7.4~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools

e.g. json-fortran@8.3.0 vs mom5@master

I couldn’t find a way get the git tag value, for example for mom5 it would be 2023.11.09.

harshula · 23 February 2024 06:28

Please use a new topic. Did you use the spack.yaml to install the packages? This is how it should look:

$ spack find
-- linux-rocky8-x86_64 / intel@19.0.5.281 -----------------------
access-om2@git.2023.11.23=2023.11.23    mom5@git.2023.11.09=2023.11.09
cice5@git.2023.10.19=2023.10.19         nci-openmpi@4.0.2
cmake@3.24.2                            netcdf-c@4.7.4
datetime-fortran@1.7.0                  netcdf-fortran@4.5.2
gmake@4.4.1                             oasis3-mct@git.2023.11.09=2023.11.09
hdf5@1.14.1-2                           parallelio@2.5.10
json-fortran@8.3.0                      pkgconf@1.9.5
libaccessom2@git.2023.10.26=2023.10.26  zlib@1.2.13
==> 16 installed packages

Topic		Replies	Views
ACCESS-OM2: Release Information ACCESS-NRI Releases release , access-om2 , model	2	304	12 April 2024
Initial planning meeting 21/10/22 modules	1	196	21 October 2022
Issues for https://github.com/coecms/oasis3-mct Earth System Model access-esm-release , oasis3-mct	2	71	7 March 2024
Audit the compiler flags used to build access-om2 on gadi with the Intel compiler COSIMA	1	220	20 February 2023
ACCESS ESM 1.5 GitHub source repositories Earth System Model github	1	108	23 January 2024

How to build ACCESS-OM2 on Gadi

Instructions for general users on Gadi v3

Installation instructions for model developers on Gadi v7

dev-build instructions for model developers on Gadi v6

Related Topics