Hi @Aidan, just wondering why /g/data/access/spack/modules/spack/config installs to /scratch?
setenv SPACK_USER_CACHE_PATH /scratch/$project/$user/spack_user_cache
setenv SPACK_INSTALL_TREE /scratch/$project/$user/opt/spack/
I guess it’s for testing, but we’re linking ACCESS-OM3 to these libraries, so it’ll break after 100 days…
See Spack libs shouldn't be in /scratch · Issue #18 · COSIMA/access-om3 · GitHub
Yeah, that probably isn’t a great design decision.
The whole thing needs a bit of a re-think. I’ll reply on this topic shortly**
** when not deluged
Why not have this in /g/data/access/spack/modules/spack/config instead?
setenv SPACK_USER_CACHE_PATH /g/data/$project/$user/spack_user_cache
setenv SPACK_INSTALL_TREE /g/data/$project/$user/opt/spack/
The simplistic answer is that not all projects have /g/data allocations. Or they didn’t used to. That might have changed.
I’m not sure if there aren’t other ways to do this entirely, but yes, as a first step that is a good change to make.
I made an attempt here, but then hit an error I didn’t understand
I have changed the spack/config module to prefer using /g/data/$PROJECT/$USER/ as the location for the install tree and cache. I guess the cache could stay in /scratch … hmm … but I changed them both for the moment.
When I did a fresh clone of spack and checked out releases/v0.19 I could successfully build zlib:
$ spack install zlib
==> Installing zlib-1.2.13-qbwlozg4cobphkvyaizqmi47xw3ara5o
==> No binary for zlib-1.2.13-qbwlozg4cobphkvyaizqmi47xw3ara5o found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/b3/b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'edit'
==> zlib: Executing phase: 'build'
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.13-qbwlozg4cobphkvyaizqmi47xw3ara5o
Fetch: 0.12s. Build: 16.84s. Total: 16.96s.
[+] /g/data/tm70/aph502/opt/spack/linux-rocky8-cascadelake/intel-2021.6.0/zlib-1.2.13-qbwlozg4cobphkvyaizqmi47xw3ara5o
It’d take me longer to figure out why your approach didn’t work, but if you’re happy to start from scratch that should work.
Aside: make sure your first spack invocation is on a login node if you’re starting from scratch, as it needs internet access to bootstrap itself.
Thanks @Aidan. On second thoughts, if I want to make an executable for widespread use (e.g. access-om3) that lives in ik11, it would be better for its spack dependencies to live there too, say in /g/data/ik11/spack (NB: doesn’t involve $user) so that users only have to join ik11 to run it.
Would that be possible somehow?
Most certainly. The point of the spack/config module was to set up some shared config, but we can always set up something especially for your use case.
I’m afraid I have limited resources this week (sick + very busy), but I do have getting a better development environment/process for ACCESS-OM3 high on my TO-DO agenda, and there are some options I’m thinking about.
Maybe the best option is to actually start a topic about this for ACCESS-OM3 (or an issue on an appropriate repo). Would you be willing to do this? Outline what you want/need to achieve, what you’ve done so far and where it is failing?
If you did make it a topic I’d suggest putting the topic in the COSIMA category. We can create a sub-category of ACCESS-OM3 if you think that is useful.
Thanks @Aidan, with your fix I now have an executable that doesn’t depend on scratch.
ACCESS-OM3 issues are here, and that’s probably a better place for discussion than the forum. Good to hear it would be possible to build dependencies into /g/data/ik11, but the top priority at this stage is to fix the spack modules that don’t update LD_LIBRARY_PATH, which makes it impossible to run the executable via payu.
Get well soon!
So I don’t forget it, there are cases were LD_LIBRARY_PATH has to be set, notably for packages that are not compiled by spack. This PR is an example of this for reference
For anyone interested, I’ve set up a new Spack instance on Gadi to be used by the COSIMA community. It reuses several things from @Aidan 's config, but I’ve tried to improve a bit on the design and make it a better fit for our use case. You can find all the config files here:
The containerised spack environment I’ve been working on is now available at ScottWales/spack-environments (github.com). This is a mirror since we’ve been heavily using Gitlab CI for the setup.
With bind mode MPI set up performance of the container matches an uncontainerised environment, with the advantage that the container is entirely self-contained and portable to other systems
Thanks for taking the time to make a public version available for the wider community. Much appreciated.