I have started to set up a shared spack installation in /g/data/access. One of the motivations to do this now was to lower the barrier to collaboration and to have a common basis from which to work: spack is very sensitive to how it is configured when it comes to find packages and determining what to install.
I’ve created a spack directory in /g/data/access/apps:
$ ls -lh /g/data/access/apps/spack
total 20K
dr-xr-sr-x+ 9 aph502 access.admin 4.0K Dec 1 17:13 0.18.0
dr-xr-sr-x+ 9 aph502 access.admin 4.0K Dec 1 13:58 0.19.0
drwxrwsr-x+ 5 aph502 access.admin 4.0K Dec 2 15:19 config
-rwxrwxr-x+ 1 aph502 access.admin 672 Dec 1 13:57 get_spack
drwxrwsr-x+ 3 aph502 access.admin 4.0K Dec 2 15:05 opt
Thought it isn’t strictly necessary to support more than spack/0.19.0 this capability will be required so I’ve started with the two most recent versions as a proof of concept.
Each spack version is a shallow clone of just the branch corresponding to the version tag. The get_spack script makes this process reproducible and relatively painless:
$ cat /g/data/access/apps/spack/get_spack
#!/bin/bash
# Copyright 2022 ACCESS-NRI and contributors.
# SPDX-License-Identifier: Apache-2.0
help() {
cat <<EOF
Download a specific version of spack using shallow clone
Usage:
get_spack <version>
General Options:
-h: Print help
EOF
}
if [ "$#" -ne 1 ]; then
echo "Must specify a version tag"
help
exit 22
fi
# version supplied as a command line argument
VERSION=$1
# Prepend v to version to create spack compatible version tag
VERSION_TAG=v${VERSION}
# Make a shallow clone of only this version
git clone https://github.com/spack/spack.git --depth 1 --branch $VERSION_TAG $VERSION
# Remove write permissions
chmod -R a=rX $VERSION
Note that the script removes write permission, as the goal is to not write into this directory.
I’ve created spack loadable modules in /g/data/access/modules/spack:
$ ls -lrtha /g/data/access/modules/spack
total 16K
drwxrwx---+ 69 wml548 access.admin 4.0K Dec 1 10:26 ..
-rw-rw----+ 1 aph502 access.admin 56 Dec 1 17:18 .modulerc
lrwxrwxrwx 1 aph502 access.admin 7 Dec 1 17:20 0.18.0 -> .common
lrwxrwxrwx 1 aph502 access.admin 7 Dec 1 17:20 0.19.0 -> .common
-rw-r-----+ 1 aph502 access.admin 2.0K Dec 2 15:30 .common
drwxrws---+ 2 aph502 access.admin 4.0K Dec 2 15:30 .
All that is required to add a new spack module version is to run the above get_spack script in apps/spack and then add a symbolic link in modules/spack to /g/data/access/modules/spack/.common. This module file dynamically finds the version from the name of the file that links to it, and sets appropriate environment variables:
$ cat /g/data/access/modules/spack/.common
#%Module1.0
# Loads standard directory structure into environment variables
#
set help "spack build from source HPC package manager"
set install-contact "https://forum.access-hive.org.au/c/infrastructure/modules/25"
set url "https://spack.io"
set spackhome "/g/data/access/apps/spack"
set version [lindex [split [module-info name] {/}] 1]
# Cannot have conda loaded, it sets way too many environment variables to play
# nicely with spack
conflict conda
# System python has a bug when accessing lustre. Need to load a version of python that
# works, so insist on a known good version
if ![ is-loaded python3/3.11.0 ] {
module load python3/3.11.0
}
set prefix $spackhome/$version
setenv SPACK_PYTHON /apps/python3/3.11.0/bin/python3
setenv SPACK_SYSTEM_CONFIG_PATH $spackhome/config/system
setenv SPACK_INSTALL_TREE $spackhome/opt/spack/$version
# the env array is a global, so need to access with :: syntax
set project $::env(PROJECT)
set user $::env(USER)
if [ file isdirectory "/scratch" ] {
setenv SPACK_USER_CACHE_PATH /scratch/$project/$user/spack_user_cache
}
if { [module-info mode load] } {
puts stderr "spack needs to create a number of shell functions and set environment variables to"
puts stderr "work correctly. Run the following command to configure your shell to use spack:"
puts stderr ""
puts stderr ". $prefix/share/spack/setup-env.sh"
puts stderr ""
} elseif { [module-info mode unload] } {
puts stderr ""
puts stderr "WARNING! This shell will still have spack specific paths and commands loaded"
puts stderr "If this causes any issues close this shell and open a new one"
puts stderr ""
}
module-whatis "[module-info name]: $help"
proc ModulesHelp { } {
variable help
variable url
variable install-contact
puts stderr "Module [module-info name]"
puts stderr "$help"
puts stderr "Information available at $url"
puts stderr "Go to ${install-contact} for more information"
}
Some important points:
- The system
pythonversion has a bug with the lustre file system, so we need to load a known goodpython3module and set theSPACK_PYTHONenvironment variable - The
SPACK_SYSTEM_CONFIG_PATHenvironment variable is set to/g/data/access/apps/spack/config/system. This allows for system wide configuration (see the docs about configuration scopes). -
SPACK_INSTALL_TREEis set to/g/data/access/apps/spack/opt/spack/$version. This is where packages will be installed to be used by others. It isn’t intended that everyone who usesspackinstall into this location. - To load the modules you need to do
module use /g/data/access/modulesand when you domodule load spackyou get this
spack needs to create a number of shell functions and set environment variables to
work correctly. Run the following command to configure your shell to use spack:
. /g/data/access/apps/spack/0.19.0/share/spack/setup-env.sh
This is because I can’t figure out how to source the bash setup script in the module file. It is against the design principle that environment modules can be unloaded and revert the shell to it’s previous state, which this can’t do.
The site specific installation files are here:
$ ls -lah /g/data/access/apps/spack/config/system/
total 16K
drwxrwsr-x+ 2 aph502 access.admin 4.0K Dec 2 15:27 .
drwxrwsr-x+ 5 aph502 access.admin 4.0K Dec 2 15:19 ..
lrwxrwxrwx 1 aph502 access.admin 30 Dec 2 12:37 compilers.yaml -> ../spack_config/compilers.yaml
-r--r--r--+ 1 aph502 access.admin 1.4K Dec 2 15:11 config.yaml
lrwxrwxrwx 1 aph502 access.admin 29 Dec 2 12:37 packages.yaml -> ../spack_config/packages.yaml
-rw-rw-r--+ 1 aph502 access.admin 46 Dec 2 15:27 repos.yml
Note that compilers.yaml and packages.yaml are generated by find_compilers_packages.py in this repo
This is still not working, but I wanted to get some eyes (principally @Scott and @dale.roberts) on this before I go too much further down the rabbit hole in case there are major issues with what I’ve done so far.
Also just wanted to document where I had go to so far.