Hi @Scott ,
@Aidan mentioned that you built binaries via Spack using Spack’s openmpi library and then ran the binaries using system openmpi libraries. Can you please elaborate on the steps you took to force the binaries to use sytem openmpi libraries instead of the Spack openmpi libraries? Thanks!
We did this primarily for containerization - we can’t build containers on Gadi, but we want to be able to run them there with the system-optimised MPI.
There’s a public copy of our system at GitHub - ScottWales/spack-environments. The container itself has a spack-managed environment which gets activated by the container’s entrypoint.
The setup for this is at containers/spack-env/install-spack-env.sh. I create my own activate script rather than using spack activate to improve startup speed.
I want the activate script to be able to run in both bind mode with the system MPI loaded and hybrid mode with only the container’s library. This means I need the container MPI to not be in a place findable by executable’s RPATH, and for the system MPI to be ahead of the container MPI in LD_LIBRARY_PATH.
To start out with, at line 18 I find the mpi library with spack find --format='{name}@{version}' mpi. This will give the realisation of the container’s mpi, currently either openmpi or intel-oneapi-mpi (the latter is still in development).
In the openmpi case I then at line 27 move the C MPI libraries to a new directory $MPI_PATH/lib_hybrid_mpi, so that it’s hidden from RPATH, and remove the library symlinks from the spack environment view. I also create some wrappers for mpirun and orted so that they start processes in the container environment. I only move the C libraries, I keep the Fortran libraries the same as they are compiler specific. Behind the scenes the fortran libraries will call the C libraries so I just need to worry about the C version.
Finally I create the activate script. I add paths for individual packages rather than the environment view primarily for debugging to be able to see what packages each file is associated with. At line 131 I pick up the bind mode MPI path from an environment variable set outside of the container, then set LD_LIBRARY_PATH to the correct order.
Thanks for the detailed explanation @Scott. The detail is very important when it comes to replicating, and deciding what is feasible to support.