I am using this topic to document steps investigating using spack as a build-from-source package manager for gadi@NCI.
The initial steps are documented in this GitHub issue, but copied here for completeness:
Results of some preliminary testing of spack on NCI
Quick set up of spack:
git clone https://github.com/spack/spack
cd spack/
. share/spack/setup-env.sh
spack compiler find
Picked up the following by default:
==> Available compilers
-- clang rocky8-x86_64 ------------------------------------------
clang@13.0.1
-- dpcpp rocky8-x86_64 ------------------------------------------
dpcpp@2022.1.0
-- gcc rocky8-x86_64 --------------------------------------------
gcc@8.5.0
-- intel centos8-x86_64 -----------------------------------------
intel@2021.2.0 intel@19.1.3.304 intel@19.1.2.254
-- intel rocky8-x86_64 ------------------------------------------
intel@2021.6.0
-- oneapi centos8-x86_64 ----------------------------------------
oneapi@2021.1
-- oneapi rocky8-x86_64 -----------------------------------------
oneapi@2022.1.0
Can load other compiler versions and for them to be found as well
module load intel-compiler-llvm/2022.1.0
spack compiler find
which added the compiler to list
-- oneapi rocky8-x86_64 -----------------------------------------
oneapi@2022.1.0
Force finding external tools
spack external find
That picks up a bunch of system tools we’re probably pretty agnostic about as far as versions go. Mostly just need builds to be reproducible, so the less we have to install as dependencies the better
$ grep spec ~/.spack/packages.yaml
- spec: cmake@3.20.2
- spec: bison@3.0.4
- spec: cvs@1.11.23
- spec: groff@1.22.3
- spec: texinfo@6.5
- spec: tar@1.30
- spec: openssh@8.0p1
- spec: binutils@2.30.113
- spec: subversion@1.10.2
- spec: flex@2.6.1+lex
- spec: automake@1.16.1
- spec: ninja@1.8.2
- spec: pkgconf@1.4.2
- spec: gawk@4.2.1
- spec: gmake@4.2.1
- spec: openssl@1.1.1k
- spec: m4@1.4.18
- spec: autoconf@2.69
- spec: libtool@2.4.6
- spec: findutils@4.6.0
- spec: diffutils@3.6
- spec: git@2.31.1~tcltk
Can load modules and ask spack to find them
module load openmpi/4.1.4
spack external find openmpi
module switch openmpi/4.1.2
spack external find openmpi
spack external find perl
module load python3/3.10.4
spack external find python
Picked up the following:
- spec: git@2.31.1~tcltk
- spec: python@3.10.4+bz2+ctypes+dbm+ensurepip+lzma+nis+pyexpat~pythoncmd+readline+sqlite3+ssl+tix+tkinter+uuid+zlib
- spec: python@2.7.18+bz2+ctypes+dbm+ensurepip+nis+pyexpat~pythoncmd+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
- spec: python@3.6.8+bz2+ctypes+dbm+ensurepip+lzma+nis+pyexpat~pythoncmd+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
- spec: perl@5.26.3~cpanm+shared+threads
- spec: openmpi@4.1.2%gcc@8.5.0+cuda+cxx~cxx_exceptions~java+lustre~memchecker+pmi~static~wrapper-rpath
- spec: openmpi@4.1.4%gcc@8.5.0+cuda+cxx~cxx_exceptions~java+lustre~memchecker+pmi~static~wrapper-rpath
Added an un-buildable virtual mpi package in ~/.spack/packages.yaml:
packages:
# Add a virtual mpi package and set it to not be buildable, to force using available external MPI
mpi:
buildable: false
which forces any MPI dependency to use one fo the installed MPI libraries.
Check what spack will install with using spec sub-command:
$ spack spec json-fortran %oneapi@2022.1.0
Input spec
--------------------------------
json-fortran%oneapi@2022.1.0
Concretized
--------------------------------
json-fortran@8.3.0%oneapi@2022.1.0~ipo build_type=RelWithDebInfo arch=linux-rocky8-cascadelake
^cmake@3.20.2%oneapi@2022.1.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-rocky8-cascadelake
Install json-fortran dependency used in libaccessom2
$ spack install json-fortran %oneapi@2022.1.0
[+] /usr (external cmake-3.20.2-hw3xgexqsgf6bjklxwit5fxhjoz6wdst)
==> Installing json-fortran-8.3.0-zyzrvosuv4732vqp3z3grchy3rqyg5og
==> No binary for json-fortran-8.3.0-zyzrvosuv4732vqp3z3grchy3rqyg5og found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/5f/5fe9ad709a726416cec986886503e0526419742e288c4e43f63c1c22026d1e
8a.tar.gz
==> No patches needed for json-fortran
==> json-fortran: Executing phase: 'cmake'
==> json-fortran: Executing phase: 'build'
==> json-fortran: Executing phase: 'install'
==> json-fortran: Successfully installed json-fortran-8.3.0-zyzrvosuv4732vqp3z3grchy3rqyg5og
Fetch: 3.34s. Build: 17.56s. Total: 20.90s.
[+] /scratch/x77/aph502/spack/opt/spack/linux-rocky8-cascadelake/oneapi-2022.1.0/json-fortran-8.3.0-zyzrvosuv4732vqp3z3grchy
3rqyg5og
$ spack find
==> 2 installed packages
-- linux-rocky8-cascadelake / oneapi@2022.1.0 -------------------
cmake@3.20.2 json-fortran@8.3.0
Look for an existing datetime-fortran package
$ spack list datetime
==> 3 packages.
py-jdatetime py-parsedatetime r-assertive-datetimes
Not available, so make one
$ git checkout -b datetime-fortran
Switched to a new branch 'datetime-fortran'
$ spack create
==> This looks like a URL for datetime-fortran
==> Found 8 versions of datetime-fortran:[https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.7.0/datetime-fortran-1.7.0.tar.gz](https://github.com/wavebitscientific/dathttps://github.com/wavebitscientific/datetime-fortran/releases/download/v1.7.0/datetime-fortran-1.7.0.tar.gz)
1.7.0 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.7.0/datetime-fortran-1.7.0.tar.gz
1.6.2 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.2/datetime-fortran-1.6.2.tar.gz
1.6.1 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.1/datetime-fortran-1.6.1.tar.gz
1.6.0 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.0/datetime-fortran-1.6.0.tar.gz
1.5.0 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.5.0/datetime-fortran-1.5.0.tar.gz
1.4.3 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.3/datetime-fortran-1.4.3.tar.gz
1.4.2 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.2/datetime-fortran-1.4.2.tar.gz
1.4.1 https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.1/datetime-fortran-1.4.1.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 8
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.7.0/datetime-fortran-1.7.0.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.2/datetime-fortran-1.6.2.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.1/datetime-fortran-1.6.1.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.6.0/datetime-fortran-1.6.0.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.5.0/datetime-fortran-1.5.0.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.3/datetime-fortran-1.4.3.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.2/datetime-fortran-1.4.2.tar.gz
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.4.1/datetime-fortran-1.4.1.tar.gz
==> This package looks like it uses the autotools build system
==> Created template for datetime-fortran package
==> Created package file: /scratch/x77/aph502/spack/var/spack/repos/builtin/packages/datetime-fortran/[package.py](http://package.py/)
Add meta-data to auto-generated package
$ spack edit datetime-fortran
Test install
$ spack install datetime-fortran %oneapi@2022.1.0
==> Installing datetime-fortran-1.7.0-smbai7glmpfkg3poxd6jdpm5hl4zjjeg
==> No binary for datetime-fortran-1.7.0-smbai7glmpfkg3poxd6jdpm5hl4zjjeg found: installing from source
==> Fetching https://github.com/wavebitscientific/datetime-fortran/releases/download/v1.7.0/datetime-fortran-1.7.0.tar.gz
==> No patches needed for datetime-fortran
==> datetime-fortran: Executing phase: 'autoreconf'
==> datetime-fortran: Executing phase: 'configure'
==> datetime-fortran: Executing phase: 'build'
==> datetime-fortran: Executing phase: 'install'
==> datetime-fortran: Successfully installed datetime-fortran-1.7.0-smbai7glmpfkg3poxd6jdpm5hl4zjjeg
Fetch: 3.89s. Build: 8.82s. Total: 12.71s.
[+] /scratch/x77/aph502/spack/opt/spack/linux-rocky8-cascadelake/oneapi-2022.1.0/datetime-fortran-1.7.0-smbai7glmpfkg3poxd6j
dpm5hl4zjjeg
$ spack find
==> 3 installed packages
-- linux-rocky8-cascadelake / oneapi@2022.1.0 -------------------
cmake@3.20.2 datetime-fortran@1.7.0 json-fortran@8.3.0
I have pushed the package description to a datetime-fortran branch on the ACCESS-NRI fork of spack.
Next test step was to build the OASIS3-MCT coupler used in ACCESS-OM2. This is a version of the OASIS code that has been modified, but the same base code has been used in a number of projects.
There was no well versioned tarballs to use for spack, so pushed a 2.5 tag to the repo as a reasonable estimate of the status of the version.
Experimented with creating an ACCESS specific OASIS3-MCT build
spack create -n oasis3-mct-access https://github.com/COSIMA/oasis3-mct/archive/refs/tags/2.5.tar.gz
spack correctly detected this as a Makefile based build.
To edit the spack build configuration
spack edit oasis3-mct-access
Stopped at this point as the OASIS build is a bit non-standard. The Makefile defaults to making an NCI specific build
the Makefile itself is here
and it sources this NCI specific file
This isn’t a currently supported configuration for spack, which only really supports editing (filtering) static files. It should work to filter the make.nci file, but I think it would probably be better to standardise the build infra. @harshula is now working on the OASIS3-MCT build infrastructure with the aim of making it consistent with other builds.